On-call role in the chain of handling system failures: simple ops only?

I've shared with my team this mental model about the escalation and de-escalation path for handling system failures and there were different opinions about the role of the on-call in it. I wonder what do you guys think?

In essence, these are the stages of where/who handles system failures:

1. software has fault-tolerance, i.e. handles failure without effect on the clients
2. automated operations, i.e. automated scripts listening to health checks, supervisor programs, etc.; no human needed
3. on-call, i.e. humans, but they use very simple instructions and runbooks; no real heavy debugging, etc.
4. expert help, i.e. humans who deeply understand the system, can do heavy debugging, have access to sources, etc.

There were different opinions in the team on whether on-call is expected to be limited to very simple runbooks/checks. Ie. ops that are simple, but for some reason were not automated (maybe some have human judgment that is hard to automate or there was never time to automate).

For the context, this discussion was in a data infra team, which has both platform engineers, SREs; the team is setting up OSS software and also develops their own, mostly complex, distributed systems. Every engineer on the team participates in the on-call.

What do you guys think? Is it beneficial to limit expectation of on-call to very simple ops and that they call/delegate to particular expert in the team to handle other cases?

More details in the post, but most info is above, no need to read. [https://medium.com/@abaranau/mental-model-for-failure-handling-on-call-run-books-etc-719240ed6cc8](https://medium.com/@abaranau/mental-model-for-failure-handling-on-call-run-books-etc-719240ed6cc8)

https://redd.it/dm27c2
@r_devops

Questions on Cloud lock-in .

I'm about to start setting up cloud infrastructure, I wanted to hear more about cloud vender lock-in and how serious can it get?
Any past experience that you guys had to deal with that you might wanna share?
I'm looking into multi cloud setup, is it advisable?

https://redd.it/dm0vi7
@r_devops

CI/CD with AWS lambda and python

Hi DevOps,

How can i setup CI/CD with AWS lambda and Python.Requirement is - Need to run test case which written in python, once the test case is successful, It has deploy the code.

​

New to CI/CD Please help. Thanks

https://redd.it/dmiqz8
@r_devops

Frustration with lack of feedback after interviews

​

Let me start by saying that I am pretty happy with my current job. I have a good salary and my work has some bonus structure, pays all my remote coworking expenses and good medical benefits for my family. I'm 35 and have tons of work experience coming from tech support, sys-admin, some development work and ultimately I am now a Sr. SRE at a \~120 person startup. I didn't go to college but I've been mostly working in the field since age 16. I've been a linux guy since redhat 4.2 in 1997-8.

​

I've never had much trouble finding a job when I needed one in the past, and usually only interviewed at one or two places before taking a job. But after some layoffs occurred at my company earlier this year I thought I should probably start interviewing in case something great comes along, perhaps finding something as good as my current job or better without potential "running out of runway" issues.

​

I've done 3 rounds interviews and have been passed on by all of them so far. I know that isn't a ton, but it's still a bit disheartening. I don't even need the job, but I'm wondering wtf is going on.

​

For all 3 interviews I have had great phone screens with recruiter and then with a technical person or hiring manager. One company gave me a take-home project which I thought was unreasonable as timeboxed to 4 hours. I committed what I had in 4 hours and then worked the rest of the weekend on it and committed that as well.

​

Another company had me do a 1 hour videocall wherein I was given a very broken shell script (about 15 flaws in the script), the end result of which was a running microservice container then some problems in AWS with security groups and load balancers to make the service reachable. I was able to complete the excercise in the timeboxed window, perhaps even a few minutes early, but the guy kept feeding me hints that I probably didn't even need if he would just give me a few more minutes.

​

Another (huge) company I had what I thought was a great call (technical and soft skills)

​

All 3 companies 'passed' on whatever the final interview stage was (usually a full-day onsite etc), and none of them would provide me any useful feedback whatsoever like it was against their policy to do so.

​

I mean really, just give it straight. You didn't fix the bash problems fast enough, we are looking for someone with experience with X, or "we already knew we were hiring my cousin".

​

​

It's pretty frustrating out there. I'm just thankful for the job I have and hope it lasts for a long time, especially if we really have a recession coming down the pike.

​

That's it ..

https://redd.it/dmmjq8
@r_devops

What kind of experience did you have before you landed your DevOps job?

What kind of work experience did you have before you landed your first DevOps job? What type of work did you do prior, how many years experience, did you have any certifications?

I'm thinking of moving to DevOps after a few years working as a developer but not sure if it's even possible.

Thanks

https://redd.it/dmofx8
@r_devops

Vault: 500 errors connecting to Consul backend, but everything works fine?

Hi everyone. Sorry if this isn't the correct place to post this, but /r/hashicorp is a ghost town and I'm quite stuck.

I'm deploying brand new Consul and Vault clusters onto GKE using the vault-helm and consul-helm Helm charts. I'm able to get the two services up and I have 5 replicas of each running in the cluster.

I'm using Consul as my backend to Vault. Once Vault is unsealed, I'm able to log in and create secrets. They are successfully stored in the backend and if I log into other nodes they can all see the secrets that I created on the first node. I have auto-unseal configured and the nodes unseal themselves as they should. Everything appears to work just fine.

The problem is that, if I run "kubectl logs <vault pod name>", I get hundreds of these after a few minutes:

[WARN] storage.consul: check unable to talk with Consul backend: error="Unexpected response code: 500 (Unknown check "vault:10.26.0.131:8200:vault-sealed-check")"

Vault is not sealed; all of the nodes have auto-unseal configured using GCP KMS, which appears to work perfectly fine. Vault and Consul run in the same GKE cluster, on the same IP subnet, and can ping each other. I can see that Vault data is being saved in Consul, so I'm not sure what the error is trying to tell me. It doesn't seem to make a whole lot of sense.

Here's the "HA" section of my Vault config in the vault-helm Helm chart:

ha:
enabled: true
replicas: 5

# config is a raw string of default configuration when using a Stateful
# deployment. Default is to use a Consul for its HA storage backend.
# This should be HCL.
config: |
ui = true

listener "tcp" {
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_disable = 0
tls_cert_file = "/vault/userconfig/cert-file/cert.crt"
tls_key_file = "/vault/userconfig/cert-key/cert.key"
}
storage "consul" {
address = "https://consul-consul-server:8500" (this is the name of the k8s service for Consul)
service = "vault"
path = "apps/vault/"
scheme = "http"
token = "12345678-a1b2-c3d4-e5f6-123456789012"
}

# Example configuration for using auto-unseal, using Google Cloud KMS. The
# GKMS keys must already exist, and the cluster must have a service account
# that is authorized to access GCP KMS.
seal "gcpckms" {
credentials = "/vault/userconfig/vault-auto-auth/vault-auto-auth.json"
project = "gcp-project-name"
region = "us-central1"
key_ring = "gcp-keyring-name"
crypto_key = "gcp-key-name"
}

# A disruption budget limits the number of pods of a replicated application
# that are down simultaneously from voluntary disruptions
disruptionBudget:
enabled: true

# maxUnavailable will default to (n/2)-1 where n is the number of
# replicas. If you'd like a custom value, you can specify an override here.
maxUnavailable: null

# Definition of the serviceaccount used to run Vault.
serviceaccount:
annotations: {}

# mlock prevents memory from being swapped to disk. If swap is enabled this should
# be true.
mlock:
enabled: true

Any ideas or suggestions would be greatly appreciated! Happy to gather any additional config that might help. Thanks in advance.

https://redd.it/dmh7k9
@r_devops

Being a conservative/libertarian in a bay area tech company

I am an OH based DevOps engineer with libertarian views looking to apply to a few companies in the Bay Area. I was talking to a few friends based out of the valley and reading the Glassdoor reviews of a few of those companies. Some of the reviews point to a culture of "drag Democrat political views" and "discriminate Conservative employees".

Is there anybody here that faced any discrimination for their political views in a Bay Area tech company?

https://redd.it/dmmruj
@r_devops

AWS question: is the reserved subscription retroactive?

Whenever you buy a reserved instance type with specified tenancy and AZ, will this be valid for the already existing instances? E.g. I buy an r5.large, eu-west-1a, default tenancy reserved instance subscription, will the price of the already existing r5.large instances be dropped as well? Or only of the new instances, started after the purchase?

https://redd.it/dmeqcd
@r_devops

The anatomy of DevOps process flow – close the loop to succeed

Business is a holistic system. Its core components are tightly coupled, and their interactions help us create the desired outcome and meet a specific market need. This value creation chain is a process that has to be well-defined, properly executed, and neatly measured to achieve repeatable value delivery. When your business has well-managed processes in place, you can start implementing continual improvements in service quality, delivery time, or production costs. It is clear as day that mature processes are mandatory for steady business growth. It is what ensures clarity

DevOps business processes are implemented in a way to deliver new features, bug fixes, and system enhancements to production as quick as possible. Similar to Lean and Agile practices, the DevOps process flow seeks to eliminate wasteful practices and increase value-producing activities. The promise of building quality software quicker is so appealing that it is being adopted rapidly both in startups and in old-school corporations. According to [the state of DevOps report](https://puppet.com/resources/whitepaper/state-of-devops-report/thank-you) by Puppet, companies that have adopted DevOps methodology benefit from 46x more frequent code deployment, 440x shorter commit-release cycle and 96x faster time to recovery. Such performance is achieved by employing a set of DevOps best practices that reduce re-work and remove overhead from the software development pipeline.

While a healthy [DevOps culture](https://blog.cherryservers.com/how-not-to-implement-devops-culture) lays out the foundation for DevOps adoption, well-defined business processes ensure its success. As I‘ve put it earlier, [DevOps is an intersection of culture, processes and tools](https://blog.cherryservers.com/devops-full-picture-an-intersection-of-culture-processes-and-tools). So let‘s now examine what the stages of the DevOps process flow are and how do they help businesses develop breathtaking software.

**Continuous planning - the smaller, the better**

Forget long-term business strategies that take months to be created and are often severely aimed. Continuous business planning employs the best practices of Lean and Agile methodologies to make your DevOps process flow as smooth as ever. The main idea is to plan software development in short iterations to reduce waste and create a product that your customers would crave.

As Eric Ries stated in his book [The Lean Startup](https://www.amazon.com/Lean-Startup-Entrepreneurs-Continuous-Innovation/dp/0307887898), every new business venture must start with having its core value statement defined – a promise that should drive your customers crazy, make them fall in love with you and ensure everyone a happy ending. Of course, at first, you don‘t know for sure what your target market wants, so you roll up your sleeves, build a minimum viable product and preach it to the early adopters. The goal is to test your idea in production as quick as possible and get raw user feedback, which helps you make data-driven decisions about further development. You move in short iterations releasing new features, bug fixes, and system enhancements. Each release is closely monitored, the data gets analyzed, and your plan is modified accordingly. This way, you are less likely to go astray and can reach a [product-market fit](https://pmarchive.com/guide_to_startups_part4.html) more quickly.

**Continuous integration - set the stage**

Back in the old days, code integration was a long and tedious process. The longer your team worked on a code build, the more painful it had become. Imagine the day when after months of coding, a dozen developers are trying to integrate their code into a single piece of software. Different code branches collide, bug fixing becomes obscure, and your project timeline extends obscenely. By now, you have probably missed your project deadline, and your team is demoralized. That’s integration hell.

Continuous integration helps you get to the integration heaven, making software

integration a trivial task – you no longer think much of it. How does it work? The mainline of your code lies in the version control system. Before you start working, you make a local code copy from the repository. You then make changes to the production code and automated tests – continuous integration assumes that the lion‘s share of your code is covered by tests. After you finish your work, you create a local code build that gets tested automatically. If the local tests pass, you are allowed to commit your code to the mainline code flow. Then a new code build is created on the continuous integration server where automated tests are rerun to detect potential artifacts that may be caused by the developer‘s local environment. If these tests pass, then and only then your work is done. If things go wrong, fixing the broken build becomes the highest priority task, which, if you have implemented DevOps processes correctly, shouldn‘t take more than 10 minutes. This way, your team is confident that everybody is working with the latest code build.

Continuous integration augments continuous business planning and extends your DevOps process flow. You start with planning small, code in tiny batches – typically no bigger than 20 lines of code – and build on every code commit. This way you know if your code compiles and the initial tests pass. And when they don‘t, you spend much less time debugging the problem, since you have made just so many changes to the system. The team never gets too far from the stable code base and becomes capable of employing the next step of the DevOps process flow.

**Continuous delivery - spread chunks of value daily**

Ever since the [Agile manifesto](https://agilemanifesto.org/), there was a notion of developing software in small batches, but the deployment pipeline still lacked efficiency. Continuous delivery is the next stage of the DevOps process flow, whose primary purpose is to optimize the throughput of the [deployment pipeline](https://martinfowler.com/bliki/DeploymentPipeline.html). While the deployment pipeline starts with continuous integration, there is more to it.

The main idea of continuous delivery is to get fast, automated feedback on the production readiness of your software, every time you make a code commit. Continuous delivery ensures that new features, configuration changes, bug fixes, and experiments flow through your deployment pipeline safely and quickly in a sustainable way. Comprehensive tests take time to complete, yet you still want to make low-risk releases. So how do you balance this?

Continuous delivery goes further than merely compiling and unit testing your code. Typically, you would put more protracted and more expensive tests further away in your deployment pipeline. As a rule of thumb, these tests are also less likely to fail. The final testing suite would depend on the complexity and maturity of the system, which may require additional tests. Later stages of the deployment pipeline may include integration, load, UI, and penetration tests to prevent any performance, usability, and security issues. If the system requires thorough testing, additional tests can be executed on separate machines in parallel. This way, you get continuous feedback about the quality of your code as fast as possible, without lowering the pace of software delivery.

Depending on the maturity of your DevOps processes, continuous delivery may extend into continuous deployment. The only difference here is that with the former you keep your mainline code ready to be released at any time, while with the latter you deploy to production automatically with the condition that all the tests have passed. This way, your DevOps process flow becomes even more agile. Continuous deployment may not be suitable for financial or mission-critical applications that require extensive testing and manual intervention. If that’s not the case for you, start your DevOps journey with continuous delivery and move to continuous deployment when your DevOps processes have matured.

**Continuous operations - scale out to th

e world**

Now we step into the world of IT operations. Here, downtime is not an option. The performance of your system cannot suffer from ongoing releases, updates, and patches which are so frequent when you adopt the DevOps process flow. Continuous operations work hand in hand with continuous monitoring to bring peace of mind for agile businesses that are building software at scale.

Continuous in operations, just like everything else DevOps, start with a centralized version control system. Everything lives in it, including your code, your database schema, your server configuration files, and anything beyond that. The main idea here is to have [reproducible builds](https://martinfowler.com/bliki/ReproducibleBuild.html) of your system: if you took a virgin machine, you would be able to recreate your system out of the box. Having all your infrastructure changes logged also helps with compliance and auditing purposes. Using version control for IT operations is also the first step to abstracting away from infrastructure and treating your hardware as code.

[Infrastructure as code](https://www.hashicorp.com/resources/what-is-infrastructure-as-code) is an IT resource management approach that defines compute, storage, and network infrastructure through source code, more precisely – configuration definition files. In conjunction with cloud computing, infrastructure as code augments the DevOps process flow by order of magnitude. Infrastructure costs shrink due to a pay-per-use pricing model, deployment speed increases because of the on-demand provisioning, and infrastructure misconfiguration risks diminish thanks to automated configuration management. Even more, infrastructure as code changes the way you think about IT operations. Instead of architecting to last, you build to fail.

DevOps fits neatly with modern cloud-native applications, making [microservices architecture](https://martinfowler.com/microservices/) the primary way to run your software at scale. As soon as you outgrow the minimal viable product phase, you shatter that monolith application into loosely coupled services and run them in isolation. By doing so, you create a highly available system that is prepared to fail. If one of your services breaks down, you kill it and spin up a new one. If a load of your system increases, you merely deploy new nodes for the specific service. And if you are deploying to production daily – which you should – microservices architecture and the right DevOps tools can facilitate your deployment strategy. It becomes quicker and easier to introduce system changes gradually through canary releases or blue-green deployments.

**Continuous monitoring - "with a thousand eyes, and one"**

Let’s assume you have already implemented DevOps processes that we discussed so far. Software flows swiftly through your deployment pipeline with code integration, testing, and deployment fully automated. You are swiftly releasing new software and deploying it to production, but now you need to make sure your new releases do not cause any performance degradation. Most bugs should have been caught by now, but some artifacts are hard or impractical to test in a staging environment. Thus, testing shifts to production and extends through continuous monitoring. So what exactly is being monitored? The data collected via continuous monitoring can be divided into primary and secondary metrics.

**Primary metrics** are set to evaluate application performance that is experienced by the end-users. First, you may want to monitor *end-user experience* directly – through network port mirroring (passively) or synthetic probes (actively) – to capture latency issues and system inconsistencies as the user interacts with your application. Next, you want to monitor *business transactions* across infrastructure tiers to make sure you are meeting your SLA. Finally, you need to have *system reports* that would consist of a standard set of metrics for each application. These reports allow you to evaluate the performance of the whole system, despite cross-application difference

s. Primary metrics matter the most since they help you understand your system as a whole and how your customers experience it.

**Secondary metrics** assess computational resources of the system to ensure there is enough resource capacity to handle the load, and to identify any bottlenecks of the system. *Runtime application architecture monitoring* – that is based on application discovery and dependency mapping – helps you better understand your system topology, service dependencies, and impacts of your changes. In addition to this, you may also want to feel the pulse of your middleware through *deep-dive component monitoring*. Secondary metrics are essential to manage your system and improve its topology.

These two sets of performance metrics are closely monitored to collect the data, understand it, identify trends, and eventually take data-driven actions. By continuously monitoring your users, systems, and network, you detect and contain any incidents. Then you respond by remediating the issues, making a retrospective analysis and applying necessary policy changes. It then allows you to predict future threats and take necessary preventative actions to harden your system. Ultimately, automation strengthens these continuous monitoring stages and enables complex if-then rules to make your system self-aware. This way, you can manage your system more efficiently, self-scale the underlying infrastructure, and make informed business decisions.

Continuous monitoring closes the loop of the DevOps process flow, giving your team feedback about its development efforts. Just like everything else in DevOps, this feedback has to be taken in small sips to help you maintain pace and allow continuous planning of your development tasks.

**On the final note**

It may take time and effort to adopt DevOps culture and implement DevOps processes in your organization, but the benefits are worth the effort. You accelerate innovation, increase efficiency, reduce failures, and enhance the job satisfaction of your IT team. [There is no magic recipe to adopting DevOps](https://devops.com/there-is-no-magic-recipe-to-adopting-devops/) – it is a journey, and, like every other journey, it starts with a small step forward.

&#x200B;

Originally posted at [https://blog.cherryservers.com/the-anatomy-of-devops-process-flow-close-the-loop-to-succeed](https://blog.cherryservers.com/the-anatomy-of-devops-process-flow-close-the-loop-to-succeed)

https://redd.it/dmgg35
@r_devops

(azuredevops) initiate release stages based on package feed view

I was thinking about making a single release pipeline with all enivornments (stages).

I have my builds published in a universal package feed (or can be a nuget feed) and the latest version from any view triggers the pipeline.

Is there a (builtin) way to put in a condition for a stage where it checks from which feed view the package came from?

The use case would be, deploy a prerelease package to staging but not acceptance, or hotfix patches directly to acceptance and skip staging.

https://redd.it/dmsttl
@r_devops

Alerts on Azure?

Newbie here trying to create alerts for quotas (cores, disks, and public IPs) in Azure.

Would appreciate any pointers on how to get this configured. End goal is to have it post to Slack for alerts.

I was thinking about using the Azure Metadata Instance Service but I’m not sure if their APIs can pull this kind of information. Other idea would be to create a simple bash script on a VM that uses the az-cli along with a web hook to Slack.

https://redd.it/dmrhxy
@r_devops

Show events on a grafana graph ?

I have a PromQL query showing metrics in a time-series graph. I also have an elasticsearch server that I have with logs. I want to create a dashboard that shows these metrics from prometheus but that can show some events (log lines according to my query) on the graph timeline.

eventually I'd like to see certain metrics relativly to some events.

I tried to draw an example, the vertical lines represent the events and the graph shows a metric.

example:

[https://imgur.com/18lxQdb](https://imgur.com/18lxQdb)

https://redd.it/dlxxgr
@r_devops

Windows Licensing for Automated Test VMs

I'm relatively new to DevOps - we are trying to get certain things automated in our development shop, one of which is the implementation of automated testing.

Most of our developers use either macOs or some flavor of Linux and our deployments are all Linux-based (mostly CentOS).

Our main product is a web application that our customers access via your standard browsers (IE, FF, Chrome), and our customers are almost exclusively using Windows.

We want to set up a GitLab pipeline that runs a suite of automated tests each night, connecting to Windows VMs.

Is there an "industry standard" method for Windows licensing related to tests like this? I know that they provide ISO's for browser testing where the license is good for 90 days, but we don't have the time to set up new VMs every 3 months or so.

I know other companies out there have figured this out, and it feels like a stupid question...I'm just not sure what the best path forward is. We don't have any kind of MSDN license since no one here in the office uses Windows.

Any advice or tips would be greatly appreciated!

https://redd.it/dmy0bd
@r_devops

I made this video about chaos engineering startup Gremlin

Not sure whether this is too trivial for this subreddit, but I thought I'd share it with you nonetheless:

[https://youtu.be/6e5qNWFRXnw](https://youtu.be/6e5qNWFRXnw)

FYI, this is not a sponsored episode, I'm just a small indie content creator interested in this.

https://redd.it/dm0e67
@r_devops

Experience attracting and hiring DevOps engineers?

I'm in a leadership role at a company that is still figuring out DevOps. We've built a mini-DevOps process run by software engineers and data scientists, but it's clear to me that we need a team that is full-time focused on speeding up/automating our deployment process through better processes and tailored CI/CD software solutions. I recently posted a position for a Sr. DevOps Engineer on our company website (and on LinkedIn, Glassdoor, etc.), and haven't gotten a single bite. For reference, my company is a major employer of technology professionals in the Salt Lake City area, and it's rare for a posting like this not to garner any interest.

Has anyone here had success getting the first few DevOps professionals into a company that's just getting off the ground in this regard? What did your postings look like? Did you use agencies? Go to conferences? Any advice appreciated.

EDIT: Since you asked, here's the posting. Hopefully this doesn't count as violating Rule 3, since I'm here to ask for feedback rather than to attract candidates directly. [https://www.linkedin.com/jobs/view/1510886802/?alternateChannel=search](https://www.linkedin.com/jobs/view/1510886802/?alternateChannel=search)

https://redd.it/dmzieu
@r_devops

Have you any experience with rsync.net storage?

Hi guys!

&#x200B;

Have anyone any experience with [rsync.net](https://rsync.net)? I need a backup location where can i store database backups, script and other sys tools.

https://redd.it/dn26f7
@r_devops

How to estimate amount of code globally in a particular language?

There are some stats in github for popular languages like Java, js, c++. I’m more interested in amount of code for tools like Terraform (HCL), ansible, cloud formation, puluni, etc. Ok if public repos only and even in particular segment/location.

https://redd.it/dn3huc
@r_devops