How to stress test Prometheus host with Avalanche

Hi everyone..


I am trying to understand and do some basic stress testing for my Prometheus server but I am having a hard time going over the results and understanding them really.


I found this - [https://blog.freshtracks.io/load-testing-prometheus-metric-ingestion-5b878711711c](https://blog.freshtracks.io/load-testing-prometheus-metric-ingestion-5b878711711c)
[https://github.com/open-fresh/avalanche](https://github.com/open-fresh/avalanche)
And it seemed like a good idea/solution....
But sadly even tho I saw prometheus scrape response time spike with a few avalanche pods running..
I am not really sure how to go about into evaluating it better..


I tried looking here or other subreddits but no luck with finding a similar thread...
Anyone who has some experience to share?

https://redd.it/fbxeus
@r_devops

I'm burnt out Ex-Amazon engineer. What are my options other than DevOps?


Hello,

A little bit of history about myself. Last year, I burnt out due to stress and had a psychotic episode. I was diagnosed with Bipolar I and I had to quit my job at Amazon to focus on myself. I also had a major depression after which lasted for 3 months. Now, I'm relatively in better shape, taking my medication and seeing a therapist.

My problem is that I don't want to do DevOps related work anymore. It was my dream job but now it doesn't interest me. I'm fed up with oncalls and dealing with meaningless configuration files and systems. Something has changed in me.

Due to my condition, I need to do a remote work. I've taken a look at all available remote job websites but all I see is DevOps and programming posts. I tried to hunt for some technical writing (I love writing documentation!) but nothing has turned out. Same for customer support jobs. I think I can do technical support engineering without oncall as I love debugging systems but they're hard to find.

I'd appreciate if you can point some direction. What can I do?

Thanks.

https://redd.it/fbr8kv
@r_devops

Looking for a List of tasks for DevOps learning

I recall a roadmap and list of tasks for DevOps or a Linux admin to do in order to be one of you cool peeps.

It was "install WordPress. Deleting it. Write an Ansible playbook to do it. Delete it. Rewrite it to deploy to Kubernetes. "

https://redd.it/fbzcwx
@r_devops

NewRelic calls it what it is lmao :P

[https://imgur.com/5Lvnehg](https://imgur.com/5Lvnehg)

https://redd.it/fbz9rq
@r_devops

Jenkins slave/master on top of Kubernetes

I am trying to create this slave, master architecture using [Jenkins/Kubernetes plugin](https://github.com/jenkinsci/kubernetes-plugin).

So this are my deployment/service files.

jenkins-deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: jenkins
spec:
replicas: 1
template:
metadata:
labels:
app: jenkins
spec:
containers:
- name: jenkins
image: jenkins/jenkins:lts
env:
- name: JAVA_OPTS
value: -Djenkins.install.runSetupWizard=false
ports:
- name: http-port
containerPort: 8080
- name: jnlp-port
containerPort: 50000
volumeMounts:
- name: jenkins-home
mountPath: /var/jenkins_home
volumes:
- name: jenkins-home
emptyDir: {}

jenkins-service.yaml

apiVersion: v1
kind: Service
metadata:
name: jenkins
spec:
type: NodePort
ports:
- port: 8080
targetPort: 8080
selector:
app: jenkins


These are screenshots for the jenkins configuration. For the IP addresses i have added the commands for getting these ip addresses, cause i wanted to show which ip address i`m using

https://paste.pics/72196977028a7838aaa25eef4a314e79
https://paste.pics/40179d0d2469833194d8501600c0a42b

So as you can see on these screen shots i can open Jenkins, also i can create Jenkins jobs but these jobs are running only on the master node. I have been following the tutorial from GitHub plugin link above.

https://redd.it/fbysm6
@r_devops

Database in the Cloud

Hi guys,

I built an app with Firebase and currently am using Firestore to fulfill my data needs. Firestore lacks decent querying capabilities though, so I am looking for another way to store my data. It needs to be in the cloud since I am using serverless functions to run my backend code and obviously a database cannot be installed on such a server. I’d like to have a NoSQL database, preferably mongo.

MongoDB Atlas gives my a shared node for free in the belgian region, which is quite nice. The only concern I have is that the free tier would be too weak on peak loads, but when I upgrade to the next in line package, it costs me €60 a month which is far too much for me right now.

I could run my own VPS on for example DigitalOcean, but then security is my own responsibility which is due to my limited knowledge of Linux/database security a substantial risk. Also I have the impression that running a server dedicated to running and exposing a mongodb database is security wise and performance wise bad practice. On the other hand, those VPSes are cheaper than anything else, like €5 a month.

In short I feel that there is a giant gap between a DIY database server and a cloud managed database server and I’m not sure which side of the gap I should go for. Am I overlooking something? Would the free tier of mongodb atlas be fine for a small startup (50k reads / 50k writes an hour on peak)? What do you guys say?

https://redd.it/fbxdh2
@r_devops

Few questions about prometheus - job definition, alertmanager, and selfsigned certs

Hello.

I am using prometheus for a while but now I am going to move it outside of docker to make it more reliable. Because of this, I have some spare time to look again into my configuration files.

Now there are my 3 questions:

\- What is the definition of a job? If I have a node exporter and cadvisor on 2 different ports running on [127.0.0.1](https://127.0.0.1) does it mean its a one job or two separate jobs? Its misleading when you can set multiple targets per job

\- Should I make alertmanager running on [0.0.0.0](https://0.0.0.0) instead of [127.0.0.1](https://127.0.0.1)? Generally speaking, are there any 3rd party integrations that could benefit from making it accessible from internet? Maybe grafana needs that?

\- I have both prometheus and node exporter (hidden from public network) on the same host, should I encrypt the connection with selfsigned certs to a node exporter that runs on [127.0.0.1](https://127.0.0.1) or this would be over engineering?

https://redd.it/fc38gh
@r_devops

The versatility of Kubernetes' initContainer

There are a lot of different ways to configure containers running on Kubernetes:

* Environment variables
* Config maps
* Volumes shared across multiple pods
* Arguments passed to scheduled pods
* etc.

Those alternatives fit a specific context, with specific requirements.

Read on https://blog.frankel.ch/versatility-kubernetes-initcontainer/

https://redd.it/fbx0qm
@r_devops

Monthly 'Getting into DevOps' thread - 2020/03

**What is DevOps?**

* [AWS has a great article](https://aws.amazon.com/devops/what-is-devops/) that outlines DevOps as a work environment where development and operations teams are no longer "siloed", but instead work together across the entire application lifecycle -- from development and test to deployment to operations -- and automate processes that historically have been manual and slow.

**Books to Read**

* [The Phoenix Project](https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/1942788290) - one of the original books to delve into DevOps culture, explained through the story of a fictional company on the brink of failure.
* [The DevOps Handbook](https://www.amazon.com/dp/1942788002) - a practical "sequel" to The Phoenix Project.
* [Google's Site Reliability Engineering](https://landing.google.com/sre/books/) - Google engineers explain how they build, deploy, monitor, and maintain their systems.
* [The Site Reliability Workbook](https://landing.google.com/sre/workbook/toc/) - The practical companion to the Google's Site Reliability Engineering Book
* [The Unicorn Project](https://www.amazon.com/Unicorn-Project-Developers-Disruption-Thriving-ebook/dp/B07QT9QR41) - the "sequel" to The Phoenix Project.
* [DevOps for Dummies](https://www.amazon.com/DevOps-Dummies-Computer-Tech-ebook/dp/B07VXMLK3J/) - don't let the name fool you.

**What Should I Learn?**

* [Emily Wood's essay](https://crate.io/a/infrastructure-as-code-part-one/) - why infrastructure as code is so important into today's world.
* [2019 DevOps Roadmap](https://github.com/kamranahmedse/developer-roadmap#devops-roadmap) - one developer's ideas for which skills are needed in the DevOps world. This roadmap is controversial, as it may be too use-case specific, but serves as a good starting point for what tools are currently in use by companies.
* [This comment by /u/mdaffin](https://www.reddit.com/r/devops/comments/abcyl2/sorry_having_a_midlife_tech_crisis/eczhsu1/) - just remember, DevOps is a mindset to solving problems. It's less about the specific tools you know or the certificates you have, as it is the way you approach problem solving.
* [This comment by /u/jpswade](https://gist.github.com/jpswade/4135841363e72ece8086146bd7bb5d91) - what is DevOps and associated terminology.
* [Roadmap.sh](https://roadmap.sh/devops) - Step by step guide for DevOps or any other Operations Role

Remember: DevOps as a term and as a practice is still in flux, and is more about culture change than it is specific tooling. As such, specific skills and tool-sets are not universal, and recommendations for them should be taken only as suggestions.

**Previous Threads**
https://www.reddit.com/r/devops/comments/exfyhk/monthly_getting_into_devops_thread_2020012/

https://www.reddit.com/r/devops/comments/ei8x06/monthly_getting_into_devops_thread_202001/

https://www.reddit.com/r/devops/comments/e4pt90/monthly_getting_into_devops_thread_201912/

https://www.reddit.com/r/devops/comments/dq6nrc/monthly_getting_into_devops_thread_201911/

https://www.reddit.com/r/devops/comments/dbusbr/monthly_getting_into_devops_thread_201910/

https://www.reddit.com/r/devops/comments/cydrpv/monthly_getting_into_devops_thread_201909/

https://www.reddit.com/r/devops/comments/ckqdpv/monthly_getting_into_devops_thread_201908/

https://www.reddit.com/r/devops/comments/c7ti5p/monthly_getting_into_devops_thread_201907/

https://www.reddit.com/r/devops/comments/bvqyrw/monthly_getting_into_devops_thread_201906/

https://www.reddit.com/r/devops/comments/blu4oh/monthly_getting_into_devops_thread_201905/

https://www.reddit.com/r/devops/comments/b7yj4m/monthly_getting_into_devops_thread_201904/

https://www.reddit.com/r/devops/comments/axcebk/monthly_getting_into_devops_thread/

**Please keep this on topic (as a reference for those new to devops).**

https://redd.it/fc6ezw
@r_devops

How do I do this Jira post request in postman?

[https://developer.atlassian.com/server/jira/platform/jira-rest-api-example-add-comment-8946422/](https://developer.atlassian.com/server/jira/platform/jira-rest-api-example-add-comment-8946422/)

​

​

I am basic authing my account in the authorization tab, i'm not sure how to apply the comment body

https://redd.it/fc3b2h
@r_devops

Check out our latest blog - An intro to cluster provisioning using Crossplane. Would love to get your feedback and questions!



Introduction:

What if you could create a Kubernetes cluster across major cloud providers like Google Cloud Platform (GCP), Microsoft Azure or Amazon Web Services (AWS)  through a resource like a Deployment or a PersistentVolumeClaim (PVC) and manage it like you manage any other Kubernetes resource? That’s what you can do through Crossplane (among many other things).

Okay, what’s with the PersistentVolumeClaim (PVC) analogy? PersistentVolumeClaim (PVC) requests a PersistentVolume (PV), which under the hood provisions a storage volume according to whatever kind of storage you specify in the StorageClass.

[Read full blog here...](https://www.infracloud.io/cluster-provisioning-using-crossplane/)

https://redd.it/fc95av
@r_devops

Introduction to Application Scheduling & Orchestration

One of the hallmarks of a cloud native application is that it features high resilience against errors while providing a number of scalability options.

This is only possible because the cloud environment gives developers the ability to deploy and manage an entire cluster of containers. For smaller applications that only have a few containers, management is not much of an issue – but as applications scale, their orchestration and scheduling drastically grow in importance.

While we have touched upon this topic in our comprehensive [guide about the DevOps landscape](https://blog.cherryservers.com/complete-overview-of-devops-cloud-native-tools-landscape), this article will elaborate more on how scheduling & orchestration work.

There are various tools that help you orchestrate application servers, taking away much of the complexity that comes with deploying a large number of containers. But before we get into that, let’s begin by explaining the essential role that containers play in the DevOps universe.

## What Are Software Containers?

As a key component of modern software development that includes microservices and DevOps, you cannot understand application scheduling and orchestration without delving into the concept of containers.

By the standard [Docker definition](https://www.docker.com/resources/what-container): “A container is a standard unit of software that packages up code and all its dependencies, so the application runs quickly and reliably from one computing environment to another.”

Simply put, a container is a small-sized, standalone package of software that includes everything required to run an application; the code and all other dependencies (such as the system tools, libraries and runtime to name a few). Its core advantage is that the small size allows you to pack a significant amount of containers onto a single computer, all running on a shared OS kernel.

Before containers, the same work was being done by virtual machines, which not only packaged application code with its dependencies, but also ran an isolated operating system. This meant that many OS kernels would run on a single server, unaware of each other. In addition to this, the entire process were sometimes managed by the host operating system.

As these virtual machines run on emulated servers, there are various difficulties related to the process. Virtual machines are often an overhead that impacts the overall system performance, causing businesses to have lower performance per dollar when compared to containers.

With containers, you only package the application code, related libraries and their dependencies. Additionally, the only operating system is that of the host computer which means that containers can communicate with the operating system directly, without unnecessary overhead.

### There Are Several Benefits To Containers

One of the biggest benefits of containers is the fact that they have simplified software deployment for developers. With the essentials packaged along with the code, it is easier for developers to know that their application software will execute, regardless of where it is deployed.

Containers are also a core part of the new application development trend known as ‘microservices.’ Instead of a stand-alone, monolithic application, containers allow you to break the application down to loosely-coupled micro-services that communicate with each other through agnostic API interfaces.

Microservices architecture can lead to a vast array of benefits, covered in our [overview of the microservices software architectural style](https://blog.cherryservers.com/from-monolith-to-microservices-the-journey-towards-a-modern-cloud-native-application).

But owing to their small size, a full-size application requires a lot of containers to run – as such, there are many moving parts that need to be managed. And this is where application scheduling and orchestration comes in.

## Application Orchestration And Scheduling

Application orchestration, commonly known as container orches

tration, is a highly popular technique utilized by development teams around the world to manage an exceedingly large number of containers.

[Devopedia](https://www.devopedia.org/) defines container orchestration as: “… a process that automates the deployment, management, scaling, networking, and availability of container-based applications.”

Container management involves a large number of tasks, such as provisioning, management, scaling and networking to name a few.

With an application with five containers, a development team may be able to manage these tasks efficiently; but a large application derives data from thousands of containers. Through orchestration, developers can automate various jobs that simplify the entire process.

An important point worth noting is that scheduling is often perceived as a part of the entire container management spectrum while some experts view it as a separate container principle.

According to [Microsoft](https://docs.microsoft.com/en-us/dotnet/architecture/microservices/architect-microservice-container-applications/scalable-available-multi-container-microservice-applications), *“Scheduling means to have the capability for an administrator to launch containers in a cluster so they also provide a UI.”*

A container scheduler has quite a few responsibilities from making the most efficient use of resources to ensuring effective load-balancing across different nodes or hosts. Due to their close proximity to the cluster, they are often treated the same.

In fact, popular container orchestration tools also provide scheduling capabilities.

### How Does It Work?

The first step to effectively orchestrate your containers is to identify the right tool. Notable names include Docker Swarm and Kubernetes, but we will get to them later.

First, let’s analyze how the application orchestration and scheduling process works:

* Once you have identified your orchestration tool, the next step involves describing the application’s configuration. This can be done in either a JSON or a YAML file.
* The configuration file serves an important purpose; it directs the container orchestration tool to the location where the images and the logs are stored. Furthermore, it also assists the tool with the process of how to mount storage volume and how to establish an intra-container network –this location is generally a private registry.
* The orchestration tool will further deploy these in a replicated group onto the host server. This ensures automatic scheduling of any new deployment that takes place within a cluster after checking for predefined prerequisites such as CPU memory requirements.
* Once deployed to the host, the orchestration tool ensures that the container’s lifecycle is managed using the conditions and provisions that were laid out in the configuration file.

Usually, development teams attempt to control the configuration files by deploying the same applications across a variety of testing environments before they are deployed into production.

With container orchestration tools, developers have the freedom to choose where they are deployed. These tools can be run on a variety of environments, ranging from on-premise servers and local machines to public cloud infrastructure providers.

## The Most Popular Application Scheduling And Orchestration Tools

There are quite a few application scheduling and orchestration tools that are available in the market, with each having their pros and cons. Here’s an overview of the top three that dominate the software development market:

### Kubernetes

Kubernetes has established itself as one of the benchmark orchestration tools in the software development industry. It traces its origins back to Google, starting off as an iteration to the search engine giant’s ‘[Borg project](https://kubernetes.io/blog/2015/04/borg-predecessor-to-kubernetes/).’

Additionally, it is also the centerpiece of the famed [Cloud Native Computing Foundation](https://www.cncf.io/) that is backed by computing powerhouses such as Google, Amazon Web Services, Microsoft, IBM, Intel, Redhat and

Cisco.

The hallmark of Kubernetes remains its ability to allow developers to deliver a PaaS (Platform-as-a-Service) that helps create a hardware abstraction layer while its ability to run across leading cloud platforms and on-premise servers is another plus point. This allows teams to move workloads easily across different platforms without having to invest in application redesign.

The main components of Kubernetes include:

* **Cluster:** A set of nodes typically headed by one master node. The other nodes (workers) can either be virtual machines or physical machines.
* **Kubernetes master:** Depending on defined policies, the master manages the application instances across all nodes – from deployment to scheduling.
* **Kubelet:** Each node runs an agent process called a Kubelet that derives all relevant information from the API server.
* **Pods:** The most basic unit that may consist of multiple containers located in the same host machine; each pod has a unique IP address.
* **Deployments**: A YAML object that describes the pods and the number of container instances
* **ReplicaSet**: The number of replicas you desire to run in a cluster can only be defined by a ReplicaSet. If a node running a pod fails, ReplicaSet can ensure scheduling on an available node.

### Docker Swarm

It is yet another popular orchestration tool, one that offers complete integration with Docker. Being less complex than Kubernetes, it makes for an excellent choice for developers who are just starting with container orchestration.

Simply put, Docker Swarm allows engineers to proceed with container deployments more easily and quickly due to the inherent integration with the platform. Nonetheless, [Dockers offer both](https://blogs.dxc.technology/2017/11/01/for-cloud-container-orchestration-its-all-kubernetes-all-the-time/) – its own orchestration tool ‘Swarm’ and Kubernetes – in the hope of making them complimentary.

The main components of Swarm include:

* **Swarm:** A set of nodes, usually accompanied by a master node. Each node denotes a machine, either virtual or physical.
* **Service:** Every task outlined by the administrator that is binding on the agent nodes is a service. It helps describe which container images will be utilized by the nodes and what commands will be executed in each container.
* **Manager Node**: As the name implies, the manager overlooks the delivery and the state of the swarm.
* **Worker Nodes**: The tasks distributed by the manager get picked up by the workers. Each node reports back to the master whereas the manager only keeps track of the tasks.
* **Task**: In the Docker environment, ‘tasks’ are containers that perform the commands that are outlined in the service. Once a worker has a task, it cannot be reassigned. Furthermore, if the task fails in the replica set, a new version of the task is assigned to the next available worker.

### Apache Mesos

Made in the University of California (Berkeley), Mesos has been around for longer than Kubernetes. It is famous as a lightweight application that provides developers with advanced scalability.

A typical Mesos’ can run more than 10,000 nodes – and that is excluding the frameworks it allows to evolve independently. Additionally, it provides support in a number of popular programming languages such as Java, C++ and Python.

It is important to note that Mesos only provides cluster management solutions. As such, developers have to build the entire framework to enable orchestration of a container – a popular example includes [Marathon](https://mesosphere.github.io/marathon/%5d).

Key components of Mesos include:

* **Master Daemon**: The master node that oversees worker nodes.
* **Agent** **Daemon**: Every task sent by the orchestration framework is completed by the Agent.
* **Framework**: The orchestration platform that enables it to receive resources from the cluster manager (Mesos) and sent tasks to be executed.
* **Offer**: The information pertaining to agent nodes that is sent via Mesos to the orchestration framework.
* **Task**: The work that needs to be done based o

n resource offers.

## Benefits Of Application Orchestration Tools

Ultimately, orchestration tools take on many processes that would previously keep the developers occupied; with these, resources can be dedicated to more important tasks.

Here are some of the benefits of application orchestration tools:

#### Transportability

Modern tools allow specific application components to be scaled without affecting the rest of the application.

#### Rapid Deployment

Faced with increased traffic? Orchestration tools can assist you in the quick creation of new containerized applications.

#### Improved Efficiency

By automating several core tasks, you are reducing the probability of human errors. With such a simplified installation process, your software development team experiences a rise in productivity.

#### Highly Secure

With the containerization of applications, these tools allow you to share resources without risking the security of your data.

The software development industry has quickly moved to embrace the container model as it allows them to streamline the entire deployment process. But the success of software containers has been boosted in no small part by the advent of advanced orchestration tools that allow users to automate container management.

While Kubernetes continues to dominate the industry, there are many other tools with different advantages as well. Ultimately, the right option for you depends on your requirements and what tools can meet them best.

https://redd.it/fcc09f
@r_devops

Logging all Pull Requests / Merge Requests for audit purposes

Hi guys, we are in the process of adopting CI/CD at our company and want to work using Merge Requests. This is quite a new area in our company. In order to allow people to go to production automatically, we want to use the Merge Request feature in GitLab. In the Merge Request, approvals are given and before the Merge Request the corresponding pipeline result is displayed.

For audit purposes, is it a good idea to store these Merge Requests somewhere? Because, would it be possible for someone to overwrite the git history locally, and then let someone else merge that in?

How do other devops folks in this subreddit handle auditing their Merge Requests?

Additionally, is it a good idea to store the corresponding build logs that result from the pipeline run following the merging of the Merge Request? Audit wants to see if a change was succesfull or not.

Really appreciating your input here..

https://redd.it/fce6i2
@r_devops

What sounds better to a prospective employer?

I have received a job offer today, but I had a final stage interview last Friday with a company I like more.

Which of these e-mails would sound better?

​

> May I ask at what stage is the review of my interview? I have received other offers as well and would Like to know about my application as well at your company so I can fully review my offers in order to let everyone know my next step.

​

>My job search has reached the offer stage with my other applications. Can you let me know when you are planning to update as to the status of my application, so that I can fully evaluate my options.

https://redd.it/fcbvva
@r_devops

Is there a way to make Jest reuse cache when working in a different directory?

We use Jest in our CI pipelines. The tests take about 5 minutes to complete with no cache and about 20 seconds to complete with cache.

We are using Jenkins with multi-branch so a different folder is used for basically every job that is run. I've noticed with Jest that it knows what folder you are running from and creates a unique cache dir in /tmp/jest_XX/jest-transform-cache-2e21483d6fe4693e52e5df028952xxxxxxxx

If you are running the tests from the same folder the cache is reused but if you even do so much as rename your work folder a new directory in /tmp/jest_XX/ is created from scratch when jest is run again.

This essentially means that every single pipeline run in Jenkins is not re-using jest cache so unit tests take ~5 minutes.

https://redd.it/fcedxa
@r_devops

[Issue] Unable to access vrops after DNS and IP changed

Hi guys. need some help as I have been working this issue for the last 3 to 4 days.

History: We changed location which prompt the need for new IP and DNS, so I updated the IP and DNS via following method: `/opt/vmware/share/vami/vami_config_net`

The menu allowed me to reconfigued the IP and DNS. Next, I deleted the tomcat ./work/Catalina/localhost folder then restarted tomcat.

Still not able to bring up the login spash screen for the `https://{ip}/admin`

I verified I am able to ping the ip and resolve the DNS from the vm and from my local machine.

Can someone help if they exprienced this issue before or reconfigured the IP and DNS and what else did you do?

​

Thanks ahead.

https://redd.it/fchcx6
@r_devops

What are some good companies with full-time remote Cloud Engineer type jobs that I can apply for?



https://redd.it/fccgjl
@r_devops

Where do devops engineers look for job postings online?

Apart from Indeed and similar generic job sites, if I have a devops position available, where should I post it?

\[edit\] Followup: any tips on how I reach devops engineers who may not necessarily be searching for a job, but who might be interested in the position I have open? StackOverflow jobs?

https://redd.it/fcfiit
@r_devops