Git Binary files

Hi are upgrading to azure devops server soon from TFS 2017. Once completed we will start to use GIT as the repo. One issue we are having is go to handle binary files. How do you all handle binary files?

https://redd.it/f6ha59
@r_devops

Help on CI/CD Build before Test and why

To give you more context:

I'm a "DevOps" engineer and I'm doing my best to migrate old Jenkins to gitlab ci/cd as it meets the company needs

And since we want to use Dockers I'm being involved on the new projects to start having the best practices and train the developers on how to docker, docker-compose etc

With this I started to show to developers how gitlab pipelines work and the best practices to follow (run tests - build image - push to registry - update Dev server)

Here's now my struggle, since it's only python webapps, they found (I know this) that building the image - send to resgitry and then run the unit tests inside the container WORKS (no shit) and they changed all the pipelines to: Build the container - publish the container - run the test command inside that container - change the tag to latest if the test pass (why)

It's been 2 weeks trying to explain why it's a bad practice and I just can't find the best material/explanation why it's wrong ( plus English is not my first language)

Can someone eli5 why this is a bad practice so I can present a better ' I know it works but it's wrong?'

https://redd.it/f6gwxb
@r_devops

Talk on distributed systems (Part 2)

I recently put up the first part of this talk and got a lot of support from this community. Thanks for all your support. (Prev post link: [https://www.reddit.com/r/devops/comments/f4s9l2/a\_talk\_on\_distributed\_systems\_in\_college/?utm\_source=share&utm\_medium=web2x](https://www.reddit.com/r/devops/comments/f4s9l2/a_talk_on_distributed_systems_in_college/?utm_source=share&utm_medium=web2x))

Here is the link to part 2: [https://youtu.be/T5T0DIaWdxQ](https://youtu.be/T5T0DIaWdxQ)

Hope you all enjoy watching it!

https://redd.it/f6aqft
@r_devops

Will I get in any trouble if I do a proposal internal to the team as an open source project? (detail in post)

The story is that

I made a proposal for a new feature of our product, but for the following reasons it did not get approved from the managers and we decided not to support it

1. It is several steps ahead of our current product in terms of technology. The feature does not support any deprecated use cases of our product and it is meant to incorporate with several new features down the road map
2. Current product already supports similar feature that is not as powerful. It provides a sub set of capability of my feature
3. Current product implementation is not elastic enough and will require re-writing or adding some services to support my feature

However it turns out as a bad proposal to the team. I genuinely feel the idea is great and I don't want it to sink. So I spent my long weekend working on it, detailing the specs and implement basic mechanics. The specs for a full deploy that supports simple protocols does not seem hard to implement. But there are some catches...

* My deployment architecture is based on the concept of our team's product
* The idea has been disclosed internally with the team
* I am copying code from stack overflow pages that I referenced when writing some other code in work, making something that looks similar to my code at work

The product of our team is technically open sourced, I can find source codes and documents of released versions and all references of my implementation can be derived from published codes/documents of our team.

That is, if I start a brand new github account and deny I own it. It will seem like a random person digging through the product docs and coincidentally came up with an idea I had. This is an option but I will prefer to leave the open source project on my Linkedin page thus it should be the last resort.

Any suggestion on this?

Top priority should be publish the feature with basic standard protocol supports and walk away without getting any trouble. Second will be keeping credits on my Linkedin page

​

Edit: Our codes are published externally under BSD-3 license. But I'll like to do MIT

​

Edit:

What I mean by "made a proposal" was showing documents I wrote at home with some drawing on whiteboard. Everything hosted on GitHub and not downloaded to my company laptop and I can guarantee all commit time stamps up there are not my work time

https://redd.it/f6f9w7
@r_devops

Which 'traditional/mainstream' certs do you recommend (NOT DevOps/cloud based)? For work permit purposes

Hear me out where I'm coming from. Overseas immigration authorities when looking at eligibility for work visa first look at things like degrees incl. post-grad.

Then they look at professional qualifications by respected trade bodies.

IT certs like Microsoft, CompTiA even Cisco come even further down the list. Maybe because of lack of awareness or some other reason. I think most countries don't even give a single point for them. In fact, many countries look for "graduate" or "specialist" work. Because you can do DevOps or IT infrastructure without a degree, it's hard convincing them otherwise without some highly respected professional qualification. Salary aside, we may be considered to be "low skilled workers"!

If you wanted to future proof your career with respect to working overseas, what kind of qualification would you study on the side? I'm talking about something that takes years to achieve, not a cert you can cram in a week and pass multiple choice.

I don't think CISSP as I'm not committed to infosec. I'm thinking maybe something like PMP which is complementary and useful, especially the older I get and possibly less technical might become especially if moving into leadership.

What are you studying for on the side?

https://redd.it/f695gx
@r_devops

One pipeline to rule them all. Why?

I heard multiple companies, in recent months, say they are working on a single pipeline (or pipeline as a service) for the entire organization. My question is why is this a thing? Is this some “best practice” people are following?

Today I heard his company is asking him to fill out a 27 question spreadsheet and they will deliver him “the perfect pipeline”. This sounds like an anti pattern to me.

https://redd.it/f665pb
@r_devops

Best way to configure default settings for kubelet v1.17.1 (using Clearlinux) ?

I need to change default kubelet cgroup driver from "cgroupfs" to "systemd" (and maybe later some do further tweaking on kubelet config).

I am not sure how to do this properly with kubelet (I am using Clearlinux), some hints/links/example config file for kubelet would be nice.

https://redd.it/f65wb3
@r_devops

What Is DevOps? What is the work of DevOps Engineer?



https://redd.it/f5q16m
@r_devops

What's the use of codebuild being used with ECR?

I'm possibly going to have a junior Devops position soon and was trying to get a grasp of all these technologies. For the mostpart things have been making sense, yet I don't understand why codebuild would be used with ECS. Since you're using container images isn't it already built?

Is the whole point just to run tests on the applications?

https://redd.it/f6kjf2
@r_devops

CI/CD optimization

Hi

We are using circleci. For each check ins, it ends up with 2 test runs. One for CI and the other for CD.

This makes sense since there could have multiple PR going in at the same time.

However, for case where there is no new change, can the test run for CD be saved if it's exactly the same run as the one done for CI?

Thanks

https://redd.it/f6k7pk
@r_devops

Kubernetes Container Config

New to Kubernetes and am struggling with the best way to deploy a third party application that already has a docker image setup. The issue I am having is setting the configuration files (there are several in different subfolders) in the docker container. Locally I ran docker compose and mounted my local disk as the configuration.


I have been looking at ConfigMaps and that seems to be a bit clunky to map anything other than a series of files. Is this the typical way people are doing this and I just live with it or is there something easier?


(Running minikube locally while I learn but this will end up in Azure eventually)

https://redd.it/f6jtnp
@r_devops

Swapping homepage URL without disturbing URL paths and subdomains

Introduction:

* I have multiple Ruby on Rails applications served with Docker nginx in the same container.
* I have a domain name **website-example.com** from GoDaddy.
* On visiting **website-example.com**, we get to the landing page.
* API is exposed from the **website-example.com/getList** URL path.
* There are subdomains like **subdomain.website-example.com**

Problem Statement:

* I am building a new React application with NodeJS, but I want **website-example.com** to point to the new app.
* I want the exposed APIs to remain accessible via **website-example.com/getList**
* I want the subdomains to remain accessible like **subdomain.website-example.com**

How would I do this?

https://redd.it/f6j6c9
@r_devops

Monitoring your AWS Network Infrastructure

Monitoring your public cloud network is an unsolved problem, such as on AWS. It is far from complete, with many loose and fragmented solutions at best. For instance, filtering workloads with high inbound traffic, with specific sessions across VPC, Classifying short-lived bursts, long-lived sessions, etc. We have put together a short questionnaire looking for some feedback on possible offerings in this space.

https://forms.gle/9BH7s8sqhfDmsj117

Audience: 1) Anyone using public clouds such as AWS, GCP 2) SREs, IT solutions provider, security professionals welcome.

https://redd.it/f5k37w
@r_devops

What's your preferred way of doing HA scheduled tasks?

We have a couple crond containers that do things like cleanups and internal report/summarization tasks. These are, of course, not HA.

I've read Google's whitepaper on [distributed cron](https://landing.google.com/sre/sre-book/chapters/distributed-periodic-scheduling/) which is fascinating but we'd rather not roll our own if possible. I've also heard of people doing most of their scheduled task workload in Lambda which wouldn't really be what we're looking for. What do you guys use for HA "cron" or other scheduled tasks?

https://redd.it/f4z7af
@r_devops

K8s & Consul - Avoid cross-DC/AZ traffic

I have huge app in Kubernetes with about 90 containers on 5 k8s worker nodes. Nodes are spread between Availability Zones in AWS, like that:

node1 - AZ-A
node2 - AZ-A
node3 - AZ-B
node4 - AZ-B
node5 - AZ-C

Also, I got many small apps which use API from that huge app. All apps use external/public host of that app, so actually all traffic between apps goes through ELB.

I can use internal Kubernetes hostnames (now we got coreDNS), but I would like to avoid cross-zone traffic, so how can I implement consul to be AZ aware and prefer traffic between pods in the same AZ, e.g.

[Diagram](https://i.stack.imgur.com/6q4C8.png)

Something similar but with HAProxy and Redis:

[https://medium.com/helpshift-engineering/saving-inter-zone-transfer-costs-in-ec2-with-haproxy-c4217ac2ca04](https://medium.com/helpshift-engineering/saving-inter-zone-transfer-costs-in-ec2-with-haproxy-c4217ac2ca04)

https://redd.it/f4yo2z
@r_devops

Amazon.. holy crap...

I mean.. I just want to set up a simple couple containers. I spent 4 hours..and still have nothing running. I have had to go through cluster and policy and role and RBAC and holy shit. I am lost in why this is so forking difficult to use? I am not looking to scale 5000 servers and 4billion people. I just want to deploy my simple docker containers in a network so they communicate together.. and make it available. That's all. WTF is all this policy shit with 100's of things, and so on.

Gotta be some sort of template for common use cases?

https://redd.it/f52tzl
@r_devops

Free CI/CD tool for local dev use...

Hey all,

Been trying to figure out how to work in some CI/CD tool chain into my projects. One project in particular is my attempt at learning some microservices. I have a monorepo, which has a UI component (react), as well as some microservices that are essentially decoupled API endpoints.

I started reading up on GitLab, CircleCI, TravisCI, etc. All of those seem to be in the cloud, and while some have free options, none seem to run locally.

By locally, I was thinking I could find something fairly easy to run as a docker container that then grabs my project from github or bitbucket, can build individual microservices, the UI, etc, bundling each in to their own containers, and can then, using my docker-compose setup, stand it up to run tests. Eventually I would like to deploy to amazon or google, but right now, I am a lone developer. I don't know that I need (or want) to run all this sort of stuff in the cloud yet. So I was thinking it's 2020, there has to be some good quality open source CICD option out there (other than Jenkins) that is a little more "today" for lack of a better word.

TL;DR I would like to learn/run a "simple" CI/CD tool locally that works with my current Github project, figure out how to build separate services in my monorepo, stand up the app, run tests, and eventually be able to deploy it to a cloud service for production use.

​

Thanks.

https://redd.it/f4ocvb
@r_devops

Biggest drawbacks to Bitnami for launching WordPress sites etc?

If you regularly are deploying various websites for PHP, Laravel, WordPress, or CMS software, maybe even JS frameworks like Node etc then please offer me your opinion:

What are the negative aspects of Bitnami specifically?

It seems weird that VMWare aquired them last year when Bitnami seems big on the AWS Amazon cloud, what do insiders think is the future of Bitnami in relation to cloud servers overall?

With Docker and Kubernetes, etc, it seems even less relevant these days. Esp. if you compare with more advanced tools like Ansible.

Even Bash scripts can be organized easily now with like Mina etc.

Idk but seems Bitnami sucks from the few times I played with AWS servers... config seems pretty limited at least on the AWS versions... what's the point?

https://redd.it/f47vx7
@r_devops

Request and response validation against Swagger spec when using SAM cli local API gateway

Hi Every one,

I'm trying to write some tests against my AWS API gateway swagger specs using `sam local start-api --host`[`0.0.0.0`](https://0.0.0.0/).

It seems the fake API gateway does not validate the request nor the response against my swagger spec.

Is there any way to achieve this? like using Localstack or any other solution?

Thank you.

https://redd.it/f6vu33
@r_devops

How do I automate deployment of Logic Apps and workflows?

Hi,

I’m using Terraform to automate the deployment and management of Azure resources.

Terraform is great, and where there are gaps, I’ve used ARM templates called via Terraform.

However, I’ve run into a problem when it comes to deploying logic apps that require complex workflows and API connections to be created. Deploying a basic logic app is easy, but applying the workflow is where I’m stuck.

I understand that Terraform is for deploying and managing resources, and not deploying code or logic configuration, so I am looking for other methods in which this can be done.

Can logic apps have their logic workflow applied during CI/CD, or is there a way in which I could leverage ARM templates to deploy the Logic App and configure its workflow?

Open to any suggestions, and keen to hear how others are doing this.

Many thanks

https://redd.it/f46o2f
@r_devops

best open source way to visualize infrastructure?

we use mainly Terraform to create our infra, fully on AWS. we wanted something to create/auto-generate a VPC graph with up 2 date resource view.

I tried cloud mapper and CloudFormer, trying to auto-generate VPC graphs with resources created within, but they all seemed not so intuitive.. anyone has experience with some tools which can do this better?

PS: not expecting something magical like humans drawing boxes themselves (you can stick a fork in the whole concept if humans can do this better) . but something decent for an auto-generated infra graphs

And please don't even mention Terraform graphs

https://redd.it/f6t6wt
@r_devops