Transaction vs Service

Hello All,

Evaluating multiple APM products. They all seem to blur the concepts of services and business transactions. How would you classify the difference?

AppDynamics seems to push that a service represents ONLY by the performance of the endpoint of a service while a Business Transaction is a collection of calls across potentially multiple services to perform a some operation for the applicaiton.

Dynatrace seems to classify a transaction as each individual call made against a service.

DataDog seems to view a service as one or multiple endpoints that work together in some way and a span is the execution of a logical unit of work within or across services.

NewRelic seems to focus on transactions which it defines as one logical unit of work in a software application.

Does anyone know of a good resource to understand the differences? If I have a group of 10 containers each running one process presenting itself in kubernetes as a service and a rest endpoint that when called will need to touch all 10 processes to complete its task: would you say the rest endpoint is a service? or is the collection of all 10 processes the service?

https://redd.it/eov452
@r_devops

Cloud cost overruns with IaC?

I’ve been talking to some folks about when they experienced sizable cloud cost overruns, and any processes or tools they used to deal with it? (notably in a software dev org)

Personally I’ve experienced setting the wrong Azure Search tier in my Terraform script and tripling my monthly bill unexpectedly.

Any war stories anyone wants to share, especially when using Infrastructure-as-Code? Any best practices on how you’ve dealt with the problem?

https://redd.it/eou0c5
@r_devops

Resources regarding cost of various IPC among microservices?

I'm having trouble finding studies that share performance profiles of REST vs RPC among microservices. Has anyone come across such information that can be shared?

https://redd.it/eon7ug
@r_devops

DevOps pipeline vulnerability scanning

I am building some prototype DevOps pipelines to demonstrate the toolchain used in a typical production work flow. I have code running through static code analysis (sonarqube) , container image vulnerability scanning with clay and OpenSCAP scanner. I am wondering how do you handle the code and container vulnerabilities inside the pipeline stages. Do you fail the pipeline stage based on a number of new bugs/cves or manual promotion based on eyeballing? It seems pretty hard to automatically fail/pass stages for vulnerability scanning without manual inspection of the scan results.

https://redd.it/eomhsp
@r_devops

Managing monolithic repositories

Hey /r/devops,

some time ago I asked here how you manage your monolithic repos with subtree splits. Especially the tag syncing was a problem I now solved with a little tool we use in our CI/CD-Pipelines.

You can find my previous question [here](https://www.reddit.com/r/devops/comments/cw3q3o/how_do_you_manage_monorepos_with_subtree_splits/).



The [monorepo-operator](https://github.com/SimonBaeumer/monorepo-operator) is written in go and can be used on linux. It can manage your repos, syncs them and allows you to perform batch tasks on all repos concurrently.

Hopefully this helps other people facing the same problems. If you have any suggestions feel free to give me feedback.

https://redd.it/ep2mrg
@r_devops

How dangerous is it to work with a server that does not have sophisticated firewalls?

I am just starting to learn all the DevOps necessary to deploy my web apps.

My AWS ec-2 are all behind a security group that only allows SSH access from my static IP home address (just got one recently). I have made one DO droplet be accessible via a password (less than 10 letters) so I can screw around with it in class and when im not home. I still didn't learn how to do what I do with security groups in AWS (IP restrictions)on DO.

How dangerous is it? Can my machine ever get internet STD's from SSHing to an instance that is open like that? I read on SO that in mere minutes of my server going on the air, I can expect thousands of brute force attacks. But how can they find me so quick? It takes google weeks to index new domains. How are attackers notified of new instances so quick?

https://redd.it/eom1qx
@r_devops

CircleCI vs Gitlab

Hi all,

I know these topics come up frequently but I havent seen a specific comparison that doesnt end up as a discussion about 50 other CI tools too :)


I'm currently using Gitlab, very happy with it overall (coming from Jenkins originally) but my new client has developers who want to use Circle (and have experience with Circle), particularly for its out of the box macOS build integration (though we can solve that via gitlab with a macstadium runner in any case)


I'm not wedded to Gitlab and I know that CircleCI is a known-good tool, but I can't see any specifically great reasons to use Circle over Gitlab. The only one seems to be Github integration, which Gitlab does quite poorly.

​

Does anyone have any compelling generic reasons for why I should use Circle over Gitlab?

​

Thanks

https://redd.it/eojm6u
@r_devops

nfstream: a flexible network data analysis framework

nfstream is a Python package providing fast, flexible, and expressive data structures designed to make working with online or offline network data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real world network data analysis in Python. Additionally, it has the broader goal of becoming a common network data processing framework for researchers providing data reproducibility across experiments. Feedbacks and Contributions are welcome!

[https://github.com/aouinizied/nfstream](https://github.com/aouinizied/nfstream)

https://redd.it/eokhi4
@r_devops

HAProxy scaling for large connection count

Of the people who are currently using HAProxy in large / high traffic scenarios what are your strategies for having large concurrent connection counts.

As far as I understand, using the IPv4 TCP connection the port number is defined through 2 bytes, yielding less than 65535 ports. I read a couple articles suggesting multiple IP addresses which would obviously increase that.

Are you load balancing your load balancers? At what point do you start thinking about that?

Do you control your own DNS and hand out different load balancer IPs?

I appreciate any articles or example setups you are running *successfully* in the wild.

https://redd.it/eoh5cm
@r_devops

Creating a new AWS account for my developers

I've been tasked with preparing an AWS environment for the developers at my company.

Essentially, we already have an existing account but there are a ton of old resources (e.g., VPCs, EC2 instances, S3 buckets) in it and no user management structure in place to differentiate between AWS administrators and developers. Everyone has the same permissions.

My current game plan:

* Nuke the existing resources. Send out a global email to all of engineering and tell them to tag any resource that needs to stay.
* Figure out the best way to lock down permissions for developers. Granularly defining each permission is not a route I'd like to go down. Is there a way to provide access to everything except user management IAM functions (i.e., creating new users, password resets, etc)?

I would appreciate any guidance on how to approach this as testing and development in AWS will begin to ramp up rapidly.

https://redd.it/eogdei
@r_devops

CD Pipeline - How much time does it take in your implementation?

Trying to understand what’s a reasonable expectation in terms of how much time does it take for you to compile,stage,test and deploy to production using an automated pipeline(including automated functional test execution).I understand there are different flavors of CD based on your practices am just trying to gauge a general idea about what’s acceptable in different organizations.

https://redd.it/ep838s
@r_devops

Question about the new AWS RDS CA

When updating the database to use the new CA, you get this popup:

"Before scheduling the CA certificate rotation, update client applications that connect to your database to use the new CA certificate. Not doing this will cause an interruption of connectivity between your applications and your database."

Does anyone know if this is required? We didn't have to import the CA originally, so why would we have to now? If it's a matter of the new root CA not being part of the trust chain of older distros, does anyone know if Ubuntu 18.04 has the new CA in its ca-certificates package yet?

Thanks!

https://redd.it/ep4hyj
@r_devops

Devops Career Ladder

Hey y'all, a, hopefully, quick question for everyone. I've seen a bunch of "how to get into devops" type posts here but was wondering about the career ladder for Devops positions. I'm new to the Devops side of things (less than 1 year, though 1 year of dev experience). What does the typical ladder look like? Is it just as simple as Devops engineer I, II, III and then into management?

Thanks :)

https://redd.it/ep7qde
@r_devops

I'm looking for suggestions on the best way to keep a collection of mission-critical services running, healthy, and up-to-date.

Our needs are relatively modest, so we rolled our own solution, but even so, the development effort involved is perhaps more than we would like if there are utilities that will do most of the work for us.

At the moment, we are running our services--which are available via known, unchanging URLs--on a few bare metal Linux boxes that have lots of ram (because the services require lots of ram). We have a Python script to deploy the latest versions of the software (and the data that the software serves) to the Linux boxes. There's also a Python script to restart services with zero downtime by bringing up a new version of a service on a different port number, reconfiguring nginx to point to the new port number, and then shutting down the old version of the service.

What the scripts do is controlled by a config file.

Unfortunately, doing all this right is more complicated than it might sound, and consequently, the scripts are somewhat complicated.

So I did some brief reading up on Kubernetes and it sounds pretty good for this kind of thing. Only I don't know if would be good for our relatively modest needs. Would it simplify our lives or make them more complicated? I can't tell.

If we used Kubernetes, I suppose it would be more easy for us to move these services to the cloud should we want to, but considering how much ram we need (i.e. several terabytes), the cost of running these services 24x7 in the cloud might be prohibitive. Also, the increased latency might not be a good thing for us.

So, maybe we could just use Kubernetes on our bare metal Linux boxes. Is that easy or hard to do? Would it be a good idea or a bad idea?

Is there something for this kind of thing that would be better than Kubernetes? E.g., Ancible? Puppet? Chef? Nomad? I tried to briefly read up on these tools, but everything I've seen is written in impenetrable vague marketese. The web page for Nomad seems the most comprehensible, but I'm still completely unclear after rifling through their web pages on whether it would make for more or less work for us.

TLDR: We have a bunch of services that we want to keep alive and up-to-date with near-zero-downtime. We currently achieve this with some non-trivial home-grown scripts on bare metal. Could we do this better or smarter? I.e., with fewer man hours devoted to enhancing and maintaining our infrastructure.

https://redd.it/epaksc
@r_devops

Deploying datadog using helm | Help

I am trying to deploy datadog using the helm chart onto a Kubernetes cluster created by Kops, it has 3 masters, then two instance groups. One of those instance groups needs to be monitored and the other should have no monitoring.

​

If I am understanding the docs right I need to add labels?

​

Another question about datadog, in the helm chart is there a param I can set to be able to see if pods are pending or crash backed off? I looked over it a few times and didn't see anything about it though it seems like something that should be supported.

https://redd.it/epbygq
@r_devops

Vulnerability scanner orchestration

I want to share my first opensource project I have published https://github.com/Mixeway/MixewayHub project is designed for IT security teams which use number of vulnerability scanners but have problem with fitting them into CICD. Mixeway integrate with Nessus, nexpose, acunetix, openvas, fortify, dependency check and more. It can configure and run scans given in universal format of REST API call and in the end provide complex result set.

I am no pro dev so code quality is rather poor but I am actively doing some refactor to make it better. Any feedback would be very appreciated.

https://redd.it/ep3exb
@r_devops

How are you setting up your configuration repo? What can I do to get the most value from a new setup, specifically for ansible configuration management?

I’m going to be assisting with a brand new setup of Ansible and want some ideas and experiences with creating a new repo structure for a CI/CD pipeline. I’ll need commits to automatically be picked up and deployed to as many as 3 or 4 environments in succession.

https://redd.it/ep93wg
@r_devops

What are some of the biggest struggles in managing your AWS environments?



https://redd.it/ep41ts
@r_devops

'Unknown error 524' on netconsole

Tried running netconsole, a Linux kernel module that sends kernel logs to a remote UDP server, and was surprised to find it wouldn’t work on my AWS EC2 instance. Received the following error.

modprobe: ERROR: could not insert
'netconsole'
: Unknown error 524

Here's a blog about how to solve it [https://www.granulate.io/a-case-of-a-broken-netconsole?id=0](https://www.granulate.io/a-case-of-a-broken-netconsole?id=0)

https://redd.it/ep30z6
@r_devops

Use cases of Python for devops role

Shout out to all Devops with Python experience.

Couple of questions:

1. Do you use Python often or is it not a necessity for devops role?
2. What are best the use-cases using python for devops from your experiences?

https://redd.it/ep0p1h
@r_devops

Elastic Stack Cluster - Secure and Configure

Hello, community.

I am sharing one video where I am explaining how you easily can create your own Elastic Stack (Elasticsearch + Kibana) cluster on 3 nodes.

For this purpose I am using 3 virtual machines(Nodes) based on VirtualBox hypervisor and preinstalled and configured Linux Ubuntu 18.04 on them, preinstalled Elasticsearch with preconfigured for the demo elsticsearch.yml file and Kibana preinstalled on the first node.

So what you will need to do to have a working Elastcserch cluster is to just download the virtual machines from the provided links in the video description and import them in your VirtualBox.

In the second part, I am explaining how you can secure your Elastic Stack cluster using the instructions in the description and how you can create restrictions for users and roles in Kibana.

Link for the video - [Elastic Stack Cluster - Configure and Secure](https://www.youtube.com/watch?v=i4T1PNQZsiY)

https://redd.it/eozyvd
@r_devops