Major outage at VictorOps!

**VictorOps Service Disruption: Actively Investigating -** Apr 19, 2020 - 17:22 MDT

[https://status.victorops.com/incidents/39nq3xqr6kt6](https://status.victorops.com/incidents/39nq3xqr6kt6)

​

Update - **Resolved (total outage - 2 hours)**

After monitoring our remediation efforts surrounding this incident, we are moving this incident to a resolved status.

Brief Incident Overview: From approximately 4:30 PM (Mountain) to 5:35 PM (Mountain) on Sunday (04.19.20), we experienced service disruption challenges that may have adversely affected the overall VictorOps end-user experience.

We sincerely apologize for any unintended inconvenience this issue may have caused.

If you have any immediate questions surrounding this incident, please contact the VictorOps Support Team at: [[email protected]](mailto:[email protected])Posted 20 minutes ago. Apr 19, 2020 - 19:17 MDT

https://redd.it/g4iksz
@r_devops

nested bind mount on efs

Is it possible?

I'm using aws efs and trying to mount --bind another directory under the efs mount.

/etc/fstab example:

```
<EFS_URL>:/ /mnt nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,\_netdev,noresvport 0 0

/source/dir /mnt/nested none defaults,bind 0 2
```

If I mount the efs on another instance, /mnt/nested appears empty.

From searching online it seems that it's possible but only if I have access to /etc/exports on the nfs server.

Is there any other way?

https://redd.it/g4eb28
@r_devops

How do you do monitoring/observability for dev/staging environment?

If you are using APM tools like DataDog, NewRelic - do you also use them for staging environment? How do you optimise performance at staging - or the task of optimization is left to the production env. as long as things are working fine?

If you use APM tools for staging/dev environment also - doesn't it increase your cost significantly?

https://redd.it/g5hw7y
@r_devops

I started linting my Ansible playbooks with Drone and wrote about my experience. Hopefully this can help someone!

If you use Ansible, you've probably stumbled across some tutorials/info from [Jeff Geerling](https://github.com/geerlingguy). In an ongoing YouTube series called [Ansible 101](https://www.youtube.com/watch?v=SLW4LX7lbvE&list=PL2_OBreMn7FplshFCWYlaN2uS8et9RjNG), Jeff briefly mentions automated [linting](https://en.wikipedia.org/wiki/Lint_\(software\)) of his Ansible playbooks.

I was curious about this, and went down the rabbit hole over the past week or so. I had never done this before and thought it was a really good topic to expand on, so I wrote a two-part series about Ansible, linting, and [Drone](https://drone.io/), on my blog.

Part 1 - [How and why you should lint your Ansible playbooks](https://loganmarchione.com/2020/04/how-and-why-you-should-lint-your-ansible-playbooks/)
Part 2 - [Linting Ansible playbooks using Drone](https://loganmarchione.com/2020/04/linting-ansible-playbooks-using-drone/)

I'm by no means an expert in this area, but just wanted to document some of my thoughts, hurdles, and configurations in the hopes that it would help someone else!

https://redd.it/g5ina3
@r_devops

How to Survive as a only devops in a company without devops culture?

Hi, I've been in a devops role for 2 years now(after graduating from college) So far I've been doing a lot of things so I have a good breadth of the field but I guess I'm missing some in depth experiences(especially after reading about other devops who do stuff like messing with kernel and other in depth stuff)

To give you some background I accidentally became devops after I applied to cloud data engineer position after college. But that was actually devops(cloud engineer) job. I learnt a lot about AWS, mostly managing data lake and data pipelines. Later I was transferred to a more of a team building web-app .I was responsible for writing some docker images but actually again most of my work was with AWS.

This brings me to 8 months ago when I was approached by recruiter for a large company. They were launching their first app to the cloud so that was an exciting project for me. Well, we designed the cloud native application and now it's deployed (only dev/test environments). These were very intense months but now I kinda have nothing to do until company decide to go for production.

Now, I'm faced with couple of problems. 1. I don't have anything to do until we go to production. And it's still not sure when we plan to go live( I made some optimizations in pipelines and Terraform in this free time) 2. Since the company doesn't have any devops culture. My manager actually expects me to do everything from being responsible for deployments to being responsible for all infra. This is a bit annoying because all day I receive requests from developers e.g.: why is the build failing? why I can't ssh into the instance, etc. This actually wastes a lot of my time. 3. In my previous job I had other devops colleagues from who I might learn however now since I'm the only one I'm not sure how to keep up with all the AWS news and also I want to learn more about other aspects( The pipelines I wrote are quite mediocre IMO but I just don't have enough ci/cd experience.

Sorry for a long post and perhaps someone will be able to relate and help me to navigate this path. I went to a devops days conference last year and it seems that this industry has almost 0 junior devops engineers. I quite like my job just struggling to navigate in this complex field

https://redd.it/g5fywk
@r_devops

yamlscript, An easy to use NPM package that lets you write and run command line scripts in a YAML format

Hello 🙌🏻,

I Hope everyone is safe out there, I have made a small NPM package that lets you run a script of commands using a YAML file, the package allows you to load the scripts to be used again, also you can define which commands run in series and which commands run in parallel, there is a log file for each run for tracking.

I would like to hear your thoughts and your opinions.

Link for the GitHub repo: [https://github.com/mohamed1refaie/yamlscript](https://github.com/mohamed1refaie/yamlscript)

https://redd.it/g5kvsn
@r_devops

Newbie to Checkmarx code scans

We are building a CICD Pipeline at work using Gitlab, Chef and Artifactory. Within Chef we have to implement checkmarx code scans but I have never used it before. Looking for the easiest/quickest method or resource to learn how to use/read checkmarx code scans.

Thanks in advance!

https://redd.it/g5hti2
@r_devops

Tutorial: How to power your GitLab Auto DevOps pipelines with Crossplane

**Introduction**

Most of the times the CI pipeline’s automated test phase requires a database to be provisioned so that the business application can be tested in its entirety.

In this guide we will provision a managed Postgres database on the Google cloud platform using Crossplane and GitLab’s Auto DevOps pipeline as part of CI.

A little bit about GitLab’s Auto DevOps, GitLab Managed Apps and Crossplane in the subsequent sections.

**GitLab Auto DevOps**

GitLab provides a feature called Auto DevOps which automatically creates CI/CD pipelines for your source code repository. It is an opinionated pipeline with reasonable defaults. The pipeline can be customized by way of a yaml configuration. Auto DevOps facilitates standardisation and consistency of the CI/CD pipelines.

&#x200B;

**GitLab Managed Apps**

GitLab provides a feature of a One-Click Install of an application onto a Kubernetes cluster connected to a repository. Applications such as Prometheus, Cert-Manager or the newly added Crossplane control plane can be installed via the Managed Apps dashboard.

&#x200B;

**Crossplane**

Crossplane is an open source multi cloud control plane. It allows you to provision cloud services the kubectl way by exposing them as Kubernetes resources by the use of CRDs and controllers. Crossplane connects to the cloud provider using service account credentials and orchestrates creation of services via the cloud provider APIs. This helps developers and administrators alike in provisioning and configuration of these services using Kubernetes resources hence, allowing a uniform interface for those working on the platform. It follows the model similar to a StorageClass and a PersistentVolumeClaim where an administrator could define the class of services available to be consumed and the developers request for a claim. [read full post here](https://www.infracloud.io/auto-devops-crossplane/)..

https://redd.it/g5hjat
@r_devops

Is it normal to grind LeetCode for a DevOps interview?

I am applying for a DevOps position (AWS shop) and I was expecting the only coding to be python scripting and Jenkins/CloudFormation. Its an entry level position so I was told it was expected for me not to know too much Jenkins (I'm pretty good at Python) - but the interviewer said that for my interview make sure they can see my screen/my IDE for the coding portion of the interview for "code review".

Is this normal? Should I be trying to learn fizzbuzzes?

Thanks

https://redd.it/g5rvg6
@r_devops

Transitioning from IT to Dev Ops

Hey guys! Currently doing my fourth year of IT Support and to be honest - I am burnt out. I have no desire to continue, looking to transition into Dev Ops. Starting to learn Python and am just looking for guidance on how to get my foot in the door? All these places I look at want "x years of experience" or this and that. It seems very hard for someone transitioning into this field to land an entry level job. With that being said, how did you land your first job in it?

https://redd.it/g5r8h3
@r_devops

Want to start a YT channel but nervous.

Hey all, I’m wanting to start a YT channel doing tutorials of various devops tools like hash ivory vault, consul, git, AWS, Anthos, k8 ect.

Just to gauge the need, who would find something like this helpful? Or am I beating a dead horse?

https://redd.it/g5qts0
@r_devops

Looking to make a Jenkins Docker Container - How do i modify it with default jobs/plugins?

Hi,

I'm looking to put together a docker container (building it from a base of the latest jenkins available lts build).

I want to modify it to include a default set of build tools like g++ etc, plugins like git plugin and then have a couple of default jobs.

What is the best way to do this? I've had a play around and can sort of get the build tools and plugins working but can't seem to get the jobs added in.

I've tried creating directories and adding my config.xml files in during the docker build process but it doesn't seem to want to play ball.

Is there a well known tutorial guide on how to do this... I've read something about using volumes to persist data... is this the way to go?

https://redd.it/g5pp7o
@r_devops

Building Infrastructure Components

Hello,

I am curious on how the process is for you to decide which technologies are you choosing in order to build an infrastructure.

Imagine you need to build a platform, so you need backend (DBs, server side logic, api brokers) and frontend. How do you search/choose which languages and frameworks are you going to use?

I will maybe need to upgrade an infrastructure or build one, so I am curious on how you guys do it.

&#x200B;

Thank you.

https://redd.it/g5ghub
@r_devops

Creating a Jenkins freestyle project to build after a PR is merged into master (specific branch)

Hey guys,

Going out on a whim here, but would like to know if this is possible? Most of the documentation or stuff on Stackoverflow regard Jenkins building when a PR is pushed.

Is there a way to ignore that portion and only build after that PR is merged?

e.g. we have the master and test branch. I checkout to the test branch and make changes and then I would submit a PR. No build runs (or at least for this specific Job). My PR gets approved, I merge to master and then my build runs.

https://redd.it/g5vsan
@r_devops

Help with Kubernetes

Hi I new to Kubernetes and i have to deploy to a cluster.

My manager has given me a config file but i do not understand how to deploy using that:

&#x200B;

This is the config file

apiVersion: v1
clusters:
\- cluster:
certificate-authority-data: <data-1>
server: <api.url>
name: <url>
contexts:
\- context:
cluster: <url>
namespace: default
user: <url>
name: <url>
current-context: <url>
kind: Config
preferences: {}
users:
\- name: <url>
user:
client-certificate-data: <data-2>
client-key-data: <data-4>
password: <password>
username: <username>
\- name: <url>-basic-auth
user:
password: <password>
username: <username>

https://redd.it/g5val8
@r_devops

( HELP) Can't deploy my MERN App to Heroku/AWS. Serving static files with Express isn't working.

I am having such a difficult time deploying my React app for the past 2 weeks. I tried both Heroku and AWS. My main issue is that my web app is left with a blank screen when my node server sends React static files to the browser. The console reads Unexpected token '<'" for chunk.js. My app only works locally if I run both my client and backend servers separately. However, I know for production, I must serve static files. I feel like I saw every tutorial/forum under the sun but cannot figure out my issue.

I made a concise post here with a little more details. I would really appreciate your help, I don't want to give up on this since this is my first React app. I'm usually fine when using plain old HTML and CSS, but this is a different scenario. Thank you!

[https://stackoverflow.com/questions/61242648/why-is-my-react-app-not-displaying-when-i-serve-an-index-html-file-with-express](https://stackoverflow.com/questions/61242648/why-is-my-react-app-not-displaying-when-i-serve-an-index-html-file-with-express)

https://redd.it/g5tged
@r_devops

How to expose kubernetes dashboard to all users within my vpc using a static DNS in AWS EKS ?

I want to expose kubernetes dashboard to multiple users who have access to my vpc, i've seen some examples using internal load balancer with external DNS but i just want to know if there are more suggestions.

https://redd.it/g5id3x
@r_devops

Splitting out docker and kubernetes manifests from main source code? Production design patterns

Hi Folks,

I'm wondering what the best production design patterns are surrounding docker and kubernetes manifest files. Are you splitting out docker/kubernetes related manifests from your project's source code in git? So, two separate projects, with their own pipelines? Or is everything in a single repo, with a larger pipeline?

 

I'm of two minds, personally. If we have a single project, everything seems to get lost, and the structure is difficult to keep on top of. Developers do what they want, put projects anywhere, and it makes it difficult to keep track of changes etc.

 

On the other hand, having to trigger downstream pipelines after compiling / testing and storing artifacts is also a hassle and depending on your ci/cd tooling, can be a little tricky and cumbersome to keep track of.

 

What are you folks doing?

https://redd.it/g5fphv
@r_devops

Companies that provide multi-cloud automation services | Help with university task

Hello guys,

I am studying business management and I have an elective course called "It Infrastructure management". I have an assignment, which is called "The benefits of multi-cloud automation services" and in one part I thought about comparing several companies that provides this service.


I came up with information from this link: [https://searchcloudcomputing.techtarget.com/definition/cloud-automation](https://searchcloudcomputing.techtarget.com/definition/cloud-automation) and it says that "There is no single cloud automation tool, platform or framework. A myriad of different offerings are available, ranging from on-premises tools for private clouds to hosted services from public cloud providers.".


However, this publication was updated on 2017 of August. I thought that there are new established companies or start-ups, which provide multi-cloud automation.

My question would be, do you know any IT companies that provide multi-cloud automation services? Or there are still no companies that could provide such service and I should skip this part in my assignment?

Thanks in advance for any help!

https://redd.it/g5lzal
@r_devops

Service discovery nightmare

My company's idea of service discovery for both internal and external-facing services is to throw a couple thousand IPs on to a public hosted zone on Route 53. There is no coordination among teams regarding who adds and deletes records, leading to quite a bit of mess. Also, there are no health checks. From a security standpoint, our biggest concern is subdomain takeover due to old records never being cleaned up.


We are almost exclusively deployed on AWS, federated across several dozen accounts. There are many cross-account services, meaning service discovery has to work across accounts.

Our team wants to implement some centralized service discovery, allowing other teams programmatic access to update records that are relevant only for their team. We are leaning towards AWS Cloud Map, but we're not sure how to expose private records in one account to the others without using some DNS relay system with bind / CoreDNS.


Alternatively, if a completely managed solution can't give us what we need, we considered using Consul.


Any recommendations for / against? Other suggestions?

https://redd.it/g5c8ev
@r_devops

replacing certain lines in files depending on dev/prod environment?

I'm working on setting up multiple tools automatically (with containers) and there are quite a few configuration files to manage.

We have a testing/development environment and a production environment, so I'm wondering if there is a way to replace certain lines in files e.g. a url for dev/prod, maybe some permissions, etc.

I'm thinking of someting like Angular's environment.ts since that's the background i come from. Is there a tool for something like this just with normal files?

EDIT: If anyone ends up stumbling upon this, ansible's template module/jinja2 is a really good solution to do this

https://redd.it/g5erc2
@r_devops