Best options to monitor Nginx

Just wondering what people are using to monitor/graph Nginx web servers. I have A grafana/prometheus setup already in use for other metrics so I'm learning toward using one of the Nginx prometheus exporters, I see two on their official exporters page Nginx metric library and Nginx VTS exporter (not sure what the difference between the two is yet or which is better). Then there are other options such as collectd/telegraph etc. I assume using a prometheus exporter may be the best route?

https://redd.it/e8slbe
@r_devops

Issues deploying Consul onto Kops Kubernetes cluster

I'm having some issues deploying Consul via the offical helm chart to a Kubernetes cluster created using Helm. I have a 3 master, 3 worker node (testing only prod will be larger) cluster, I use the helm chart along with the latest version

Command used for deployment

helm install consul-dev ./consul-helm --namespace=development

As you can see I am using a pretty basic setup - the defaults just put into a namespace.

​

Seeing this error when I describe server 0

Warning FailedScheduling 25s (x5 over 3m28s) default-scheduler 0/6 nodes are available: 6 node(s) had taints that the pod didn't tolerate.

​

Output of describing a worker node. Looks like there is a taint but I never put that there so not sure why it'd be there.

me@LAPTOP-LINUX:~/$ kubectl describe node ip-10-0-111-37.ec2.internal
Name: ip-10-0-111-37.ec2.internal
Roles: node
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=t3.small
beta.kubernetes.io/os=linux
failure-domain.beta.kubernetes.io/region=us-east-1
failure-domain.beta.kubernetes.io/zone=us-east-1c
kops.k8s.io/instancegroup=nodes
kubernetes.io/hostname=ip-10-0-111-37.ec2.internal
kubernetes.io/role=node
node-role.kubernetes.io/node=
Annotations: flannel.alpha.coreos.com/backend-data: {"VtepMAC":"ce:d4:a9:0c:25:79"}
flannel.alpha.coreos.com/backend-type: vxlan
flannel.alpha.coreos.com/kube-subnet-manager: true
flannel.alpha.coreos.com/public-ip: 10.0.111.37
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Tue, 10 Dec 2019 18:41:24 -0500
Taints: node.kubernetes.io/unreachable:NoExecute
node.kubernetes.io/unreachable:NoSchedule
Unschedulable: false
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure Unknown Tue, 10 Dec 2019 18:43:31 -0500 Tue, 10 Dec 2019 18:44:14 -0500 NodeStatusUnknown Kubelet stopped posting node status.
DiskPressure Unknown Tue, 10 Dec 2019 18:43:31 -0500 Tue, 10 Dec 2019 18:44:14 -0500 NodeStatusUnknown Kubelet stopped posting node status.
PIDPressure Unknown Tue, 10 Dec 2019 18:43:31 -0500 Tue, 10 Dec 2019 18:44:14 -0500 NodeStatusUnknown Kubelet stopped posting node status.
Ready Unknown Tue, 10 Dec 2019 18:43:31 -0500 Tue, 10 Dec 2019 18:44:14 -0500 NodeStatusUnknown Kubelet stopped posting node status.
OutOfDisk Unknown Tue, 10 Dec 2019 18:41:24 -0500 Tue, 10 Dec 2019 18:44:14 -0500 NodeStatusNeverUpdated Kubelet never posted node status.
Addresses:
InternalIP: 10.0.111.37
Hostname: ip-10-0-111-37.ec2.internal
InternalDNS: ip-10-0-111-37.ec2.internal
Capacity:
attachable-volumes-aws-ebs: 25
cpu: 2
ephemeral-storage: 125753328Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 2013532Ki
pods: 110
Allocatable:
attachable-volumes-aws-ebs: 25
cpu: 2
ephemeral-storage: 115894266893
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 1911132Ki
pods: 110
System Info:
Machine ID: <ID>
System UUID:

<ID>
Boot ID: <ID>
Kernel Version: 4.9.0-11-amd64
OS Image: Debian GNU/Linux 9 (stretch)
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://18.6.3
Kubelet Version: v1.13.12
Kube-Proxy Version: v1.13.12
PodCIDR: 100.96.10.0/24
ProviderID: aws:///us-east-1c/i-0fd2d3538b39b1952
Non-terminated Pods: (2 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system kube-flannel-ds-fwhgt 100m (5%) 0 (0%) 100Mi (5%) 100Mi (5%) 32m
kube-system kube-proxy-ip-10-0-111-37.ec2.internal 100m (5%) 0 (0%) 0 (0%) 0 (0%) 32m
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 200m (10%) 0 (0%)
memory 100Mi (5%) 100Mi (5%)
ephemeral-storage 0 (0%) 0 (0%)
attachable-volumes-aws-ebs 0 0
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Starting 32m kubelet, ip-10-0-111-37.ec2.internal Starting kubelet.
Normal NodeHasSufficientMemory 32m (x2 over 32m) kubelet, ip-10-0-111-37.ec2.internal Node ip-10-0-111-37.ec2.internal status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 32m (x2 over 32m) kubelet, ip-10-0-111-37.ec2.internal Node ip-10-0-111-37.ec2.internal status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 32m (x2 over 32m) kubelet, ip-10-0-111-37.ec2.internal Node ip-10-0-111-37.ec2.internal status is now: NodeHasSufficientPID
Normal NodeAllocatableEnforced 32m kubelet, ip-10-0-111-37.ec2.internal Updated Node Allocatable limit across pods
Normal Starting 32m kube-proxy, ip-10-0-111-37.ec2.internal Starting kube-proxy.
Normal NodeReady 31m kubelet, ip-10-0-111-37.ec2.internal Node ip-10-0-111-37.ec2.internal status is now: NodeReady

Line matching for a node in `kubectl get nodes`

ip-10-0-111-37.ec2.internal NotReady node 36m v1.13.12

https://redd.it/e8zaeu
@r_devops

Verification of database changes?

Hey r/devops,

First off, I apologize if I am on the wrong subreddit for this, I'm a "junior devops" in my company so I felt this question would be appropriate here. I also apologize for any formatting issue or vagueness of my question.

I was wondering if I might be able to get some guidance on a problem I have.

We have a Hortonworks Hive database. I'm one of the people responsible to migrating/applying database changes using scripts that the developers have given me.

They've been asking for some automated way of tracking database changes and outputting a report of what has changed after a migration.

i.e.
Table A = col 1, 2, 3

Table A after migration = col 1, 2, 3, 4

Report = Table A after migration = col 4 added. Script that applied change to Table A successfully ran.

The program has to read the script it's about to apply, apply it, then confirm whether the change successfully applied or not. They don't want a report that they have to read through, they want it to just confirm that everything worked automatically.

The real problems are these:

1. I have ideas on how to do this but they seem extremely tedious and I don't have time to do it on top of all the other stuff they're asking me to do.

2. I basically have no tools or the ability to install tools. All I have is some extremely limited jenkins and whatever bash/python comes installed with an Amazon EC2 instance Linux Redhat OS. I can't install anything else, even python libraries. (At least server wide, if it fits in my home directory, it can work.)

Is there some application of bash/python script that can help me with this? How have others done it without anything but bash and python?

I've already looked at tools like Liquibase, but my company refuses to install anything.

Sorry for the wall of text.

https://redd.it/e909tu
@r_devops

Using dynamic inventory for better Ansible provisioning with Packer and WinRM

Hey y'all. Just wanted to share this blog I wrote up. When trying to solve this issue I found some discussion on the topic here a few months ago. Figured I'd do a new post instead of digging up an old thread.

https://medium.com/@borncamp/using-dynamic-inventory-for-better-ansible-provisioning-with-packer-and-winrm-72cd393cc19f?

https://redd.it/e8vp79
@r_devops

What sysadmin skills do I need to become better devops

I started as a back end engineer and gradually transitioned into DevOps.. I feel I need better understanding of memory allocation, CPU usage and resources.

Any sysadmin here doing DevOps? Kindly advice.

https://redd.it/e8xq7x
@r_devops

Which is better: Logz.io | Elastic.co | Loggly for logging needs?

I need to store logs from my application - roughly around 5GB/day with 15 day retention. Which of the following is best suited - [logz.io](https://logz.io/), [elastic.co](https://elastic.co/) managed service or vendors like loggly, sumologic, etc? What are the pros and cons for each?

https://redd.it/e8twrh
@r_devops

Help me Dockerify this mess!

So when developing locally i use a docker-compose file that has the following services:

\- Node.js worker (performs cron jobs by polling endpoints with a small api exposed to manually kick off processes)

\- Hasura GraphQL Server

\- React admin panel - kick off crons manually and review data

\- Postgres (uses dbaas in prod, though)

\- Redis (uses dbaas in prod, though)

&#x200B;

However, I've read that using docker-compose in production is not great practice but because this is a side-project and has but a few hundred users k8s feels like overkill. I would also like to figure out a way to use Prometheus, Grafana, ELK, and either Traefik/Caddy but throwing all of this into a docker-compose file seems very wrong... Would anyone give me some advice or steer me in the right direction? How would you do it?

https://redd.it/e8ynme
@r_devops

Environment Tracing and Build Annotations with Azure Pipelines

Neil Peterson does a great job in the following article, [Environment Tracing and Build Annotations with Azure Pipelines](https://techcommunity.microsoft.com/t5/ITOps-Talk-Blog/Environment-Tracing-and-Build-Annotations-with-Azure-Pipelines/ba-p/1056276?WT.mc_id=ITOPSTALK-reddit-abartolo), in detailing how to achieve environment traceability when using Azure Pipelines. An Azure Pipeline is essentially a series of automated tasks. Many tasks are built into Azure Pipelines or are available to install from the Visual Studio Marketplace. Some Azure Pipeline tasks include traceability annotations out of the box. The Kubernetes Manifest task deploys a Kubernetes manifest file to a Kubernetes cluster. When doing so, the objects created by the manifest file are automatically annotated with information about the Azure Pipeline build.

https://redd.it/e95jtr
@r_devops

The best courses to get to industry-recognized qualifications, skills

Hi

Cards on table: I'm going to make pitch to bring in some new technical courses to a training provider, in areas where the demand for technical skills is greater than supply, where employers are crying out for people familiar with certain technologies.

So I'd like to ask this community:

1. What technical skill or skills is a hot area right now?
2. Do those skills have a recognized qualification or certification path?
3. What would be the pre-requisites to that skill being acquired?
4. What is the best course or courses which would lead to acquiring those skills?

So for example:

1. AWS SysOps
2. AWS Certified SysOPs
3. Linux skills in administration container technologies
4. Amazon's own courses?

Thanks

https://redd.it/e8xl4v
@r_devops

Software signing solution - key storage

Hey guys,
In our company someone ages ago created some "system" to sign software we publish where the private key to the certificate is stored on HW HSM and that is connected to some "USB over IP" little box so it can connect to VM.

We're now in process of redesigning the whole signing process as it currently requires too much manual work and I'm not really a big fan of this box thingy. Naturally, we do not want to plug the HSM just into one of our servers, because what to do in case of server maintenance, right?

So I wonder how others deal with this, where do you store the private key for signing certificate ?
I'm even considering using something like Azure Key Vault - does anyone have any experience with that?

https://redd.it/e974jo
@r_devops

grok pattern

What is the grok pattern to parse data from this Log?

https://i.imgur.com/FT1P489.png

see these values are preceded by some text and an equal.

What pattern can parse the data value from such log line ?

https://redd.it/e98kz2
@r_devops

How do you handle multiple environments for your tools?

Currently, I have several scenarios where I deploy a series of auxiliary tools (an instance of Gitlab, Vault, Jenkins, Nexus, etc.) in a particular subnet for a team, separate from the subnets used for the apps they are deploying. Their apps are usually isolated on a per-environment basis, but the tools aren't, I just deploy a single instance.

This means that when, for example, we upgrade Jenkins or add a plugin to it, we are running into a risk of causing a downtime or, worse, having data loss. Should all changes to the tooling part of the infrastructures be tested in separate environments, like we do with our own apps? Is there any best practice for this?

https://redd.it/e95riv
@r_devops

Lost in the Security Rabbit Hole with a side of Logging

Hello r/devops,

I am very new to this space but have been loving it so far and seeing how helpful this community is, it motivated me towards leaving my trusty shadow cave and come out of lurking.

I and my team inadvertently found ourselves having to do a ton of DevOps implementations due to the recent pivot of our company. We are a team of 4 developers with next to none operations or sysadmin experience. We've faired well until now, due to limited requirements from us. The silver lining is that we are given complete control of the DevOps infrastructure project, which is being built from scratch) and we are learning, experimenting, and implementing slowly but surely. However, we found ourselves in a bit of a rabbit hole on two subjects: Security and Logging.

We would like to get your opinions and pointers on potential solutions to these two components. A bit of background distilled into pointers:

* The infrastructure will host a financial services platform that will be used by financial institutions as well as individual investors, academics and hobbyists (Though in a limited manner than institutions).
* Domain oriented micro service architecture.
* A service will contain its required components in itself such as caching and database.
* AWS is the main service provider.
* EC2, CloudWatch, Lambda, S3, and a few others
* The main programming language is Java with heavy usage of Spring Framework.
* Sensitive information will flow in and out of the system.
* Client communications and all communications between individual components must be secured.
* We have to be able to utilize directories of our customers as a means of authentication and authorization.
* We are required to record everything (some sensitive information as well) in logs and store these logs for certain periods based on metrics defined by law. Such as:
* Logs that are less than 12 months old need to be accessible within the hour.
* Logs older than 5 years must be accessible within the month.
* Everything must be stored for 20 years (Crazy).
* A small team with a lot of responsibility means we have to go for fewer cogs to achieve the desired result. We would love to find the best solution for our problems but we prefer having a single component to achieve 80-90% rather than 8 components to achieve 100%.

**Where we are at:**

Logging and Security are not the only two aspects we haven't figured out yet but they are the next on our project plan. I will most likely ask about other components on this subreddit when we get there. In terms of these two components, our progress so far:

* We are stuck on ELK vs Graylog. ELK seems to be the go-to choice at the moment but Graylog seems easier to use and has fewer moving parts. It is safe to say we are leaning towards Graylog.
* We identified a couple of Open Source projects that offer a plethora of security-related features and we distilled the list into two options: Gluu and KeyCloak. None of us has a deep understanding of this subject as our requirements for security implementations have been very limited so far. We imagine that we will have to utilise some AWS features along with these but there is just too much stuff! This area seems like a proper rabbit hole that we end up more confused than before. Our identified requirements and thought solutions are as follows:
* Microservices among themselves. Also, microservices and their databases.
* Behind private subnets and firewalls
* No internet access
* Services and API Gateway (Haven't decided on the gateway yet).
* Public gateway accesses Public Load Balancer
* Public LB forwards to firewall and intrusion detection and prevention (ID/P) layer
* If all is well, firewall - ID/P layer forwards request to private LB
* Private LB authenticates the request via Gluu/Keycloak (That resides on a separate environment)
* Customers and API Gateway
* Gluu/Keycloak and Gateway features?
* Gluu/Keycloak
* Behind a public LB and firewall - ID/P laye

r?
* Access to Customer LDAP (or similar)
* Private VPN between Gluu/Keycloak and customer system?

**What you can help with:**

Of course, if you have any comments, questions, pointers or answers to anything you see here, you are of course more than welcome to do so. However, our list of unanswered questions are as follows:

1. We couldn't find a solution to fit our log archival requirements and we decided on custom building it into our job automation system. Did we miss something or is this actually the case?
2. Is our understanding of Graylog vs ELK accurate? Unless we grossly overlooked a very important aspect in our comparison of the two, we will be going with Graylog. What is your opinion on this?
3. How would you go about implementing a security solution for a system like this? Does this seem like overkill or is it appropriate?

Thank you for taking the time and to read this far!

https://redd.it/e96365
@r_devops

Any opinions on fastly vs cloudflare?

A bit of context: mid-size startup, currently evaluating CDNs that can support some fancy logic on the edge. Lambda@Edge seems logical given that we're mostly on AWS, but after just paging through its documentation I had a distinct impression of a clunky, half-baked product. This leaves Cloudflare (with their Workers) and Fastly (they have Webassembly on the edge in their labs). We push a few TB per month out, ever increasing, so we're probably looking at Enterprise pricing from Cloudflare and it's in low $1000s/mo, which is OK. However, we didn't have any experience with neither. From my perspective, cloudflare seems a bit more hip and trendy; fastly is apparently older and less likely to get bought out or suddenly change. What do you think folks? What are the pros and cons of going with one or another?

https://redd.it/e9cbms
@r_devops

What does a DevOps (practitioner/engineer) do during a "Normal" day of work?

Probably an incredibly silly question but I am just trying to envision and understand what the actual job looks like....

What tools do DevOps folk use? Any IDEs?

What problems are they solving on a daily basis?

Do they code? If so what languages? And what are they coding?

Just trying to get a deep thorough understanding of what the job itself really looks like in terms of tasks, actions & daily chores? I would really appreciate any DevOps folk to describe any parts of their day to day work?

Also sneaky extra question... What is the difference between a DevOps practitioner & a DevOps engineer?

Thanks for your help

https://redd.it/e94huc
@r_devops

Cant join worker nodes in Kubernetes on V1.17

Im using a cluster I created on 1.16.3 and worker nodes are stuck at NotReady... I created additional nodes but they never show to Ready state..it looks like they are on 1.17 I dont know if its some compatability issue

&#x200B;

kubectl describe node shows :

Ready False Wed, 11 Dec 2019 07:44:31 +0000 Wed, 11 Dec 2019 07:35:27 +0000 KubeletNotReady runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

&#x200B;

I tried upgrading the master control plane to 1.17 and one node joins but the other gets stuck. IT looks like I only have two worker nodes max Im trying to have 3.

https://redd.it/e9446v
@r_devops

What's recommended for homebrew tek into employment?

I'm getting started on my DevOps homebrew teks soon. It has taken me awhile to get into it. I have the book by Sobell "A Practical Guide to Linux" which is also my college textbook. I'm taking "Advanced Linux" and "Intermediate Python Programming" at local college to aid my resume and have skill. Ok so I have LinuxAcademy am thinking I will need Git, Docker, Kubernetes, Jenkins, Ansible, etc. Pretty cool I have everything I need also I put in my i5 quad core laptop 16GB of RAM so I can actually simulate things like spinning up instances either locally or on the cloud I want to embrace AWS I have my "free tier" account used it for the Python development course on LinuxAcademy. Basically if there is anything recommended for employment I'd take advice. My attempts at employment lead to a lot of rejection finally after questioning reality I've decided to make strides at skill and having talent would serve me well. Like for example, I could come off like I enjoy the work ask more questions, and show the employer what I can do hands on with the laptop at their office. I'm casual about things but want to make sure my path leads to a bulls eye mark of hire.

Python and GO are on the to do list. Ruby might be useful for Vagrant and such but idk if I can master 3 languages. I want to do artifacts of SCMs like Git and be really sophisticated at this very devoted and able to make a good paycheck. Thanks! Want to do Terraform.. Have heard of Perforce and other tools too. Basically if there are a list of tools that do artifacts and else I should be learning/be expert at please let me know here..Like what all do I need of Git... I can do checkouts and merges. I did a cherry picking exercise with a simulator as well from a Stanford friend. Thank you!

https://redd.it/e92l3z
@r_devops

How does one actually microservice?

This feels like a weird question in almost-2020 but it’s serious.

You can find countless videos and blog posts on why to do the micro services and what high level architecture looks like.

There’s lots of discussions around integration technologies, synchronous vs asynchronous communication, CQRS, circuit breaking, service discovery.

What I can’t find is how people _actually_ implement these architectures in the real world.

How do people handle different stages?

Do services (including the client applications) just always consume the prod version of other services?

Do you build a giant service-stage matrix to keep track of what stage of a service you communicate with?

How are you handling data that needs shared across different services?

What do you do when a new service needs historical data from an older service?

What do you do when inevitably build too granular of services and need to merge some together?

How to handle multiple auth needs, such application to service versus service to service?


I just have so many seemingly basic questions that I can’t find good answers to and I’m losing faith that people actually implement these kinds of system in the real world. Does anybody have some good pointers for me here?

https://redd.it/e9gz44
@r_devops