AWS security, compliance + efficiency



Trend Micro recently acquired the ability to scan, report on, and auto correct (if so desired) cloud accounts against both the “Amazon Well Architected Framework” and “NIST 800-53”. In just a few clicks, you can have complete visibility across your cloud workloads and we can help you stay compliant in an ongoing manual and/or automated fashion.

**A demo of our new cloud configuration solution, Cloud Conformity, takes under 30 minutes. Anyone interested in a call next week?**

https://redd.it/e8fagx
@r_devops

Vault OSS - Backups?

Just trying to get my head round deploying vault and getting some form of DR in place. We have 3 nodes in 3 AZs and three Consul nodes providing backing. From my understanding Consul provides a snapshot capability that we can use to snapshot the storage for a point in time of the store. Reading around some of the mailing lists I can see that there is a discussion that restoring from a storage backup may lead to vault being in an inconsistent state.

I am struggling to get my head around the process for recovering from a total vault/consul outage - essentially an etch-a-sketch end of the world where we no longer have vault nodes or consul nodes. Would it be a case of spinning out new clusters of each and then restoring the consul snapshot to the consul nodes? I feel that there is sufficient HA to cover the majority of failures but would feel happier if I knew we could recover the whole thing if needed. Or is it that the key/value store is not that important to recover as vault can regenerate but the policies we wrote on the vault server are more critical to recover?

https://redd.it/e8ptyk
@r_devops

How are you supposed to provision RDS when using Elastic Beanstalk & Terraform?

I'm setting up a Rails app with Elastic Beanstalk & Terraform.

The app requires some initialization in the database, some seeding & such.

I don't understand how i am actually supposed to do this.

- There is apparently no way to specify container_commands through Terraform.

- I was thinking of running a local_exec provisioner & importing a `.sql` file, but that would require the database to be exposed publicly or having some kind of bastion ec2 instance which seems overly complicated.

- There might be a way to use a launch script, but i dont know how you are supposed to tell it to only run on the initial deploy & on one machine.

Should i just not use Beanstalk? This seems like a very trivial problem and i cannot find any info on solving this.

https://redd.it/e8e2pb
@r_devops

What role does Enterprise Architecture play in your DevOps team / organisation ?

What role does EA (Enterprise Architecture) play in your DevOps team or organization? It's a hot take for some time now that EA might be redundant as design can be done by teams themselves, or that they should provide some guidelines based on money / business requirements.
However in other organizations they still have an iron grip on every piece of design and using Architecture documents enforce old fashioned standards on teams.
Some try to create more deliverable architecture, use documentation in code and provide shield for audits to have a broader relevance in DevOps.

In any case, the field is changing and different approaches to it are taken. I was curious to hear about your opinion and / or your organizations stance.

https://redd.it/e8srtj
@r_devops

Is anyone using eBPF for monitoring?

Did some research the past few weeks on monitoring startups using eBPF (link below).

Outside of a few large tech companies (Facebook and Netflix in particular) and some startups building products *using* eBFP, is anyone using actually using it for system, network, or application monitoring? If so, what has your experience been like?

My post on startups using it + some background:

[https://monitoring2.substack.com/p/ebpf-a-new-bff-for-observability](https://monitoring2.substack.com/p/ebpf-a-new-bff-for-observability)

https://redd.it/e8tqu7
@r_devops

Would Love Some Help Translating This IBM Listing

I've been deep in Python for a while now, and I'm growing more confident in my abilities to code, but I know I'm still a beginner. I've been trolling around the internet looking for good entry-level jobs where I could get both experience in the field and some professional mentoring from seasoned programmers. This lead me to IBM, where I found half a dozen entry-level openings for devs and enigneers. One of them is a DevOps position located here:
[https://careers.ibm.com/ShowJob/Id/717577/Entry-Level-DevOps-Developer-Engineer/?lang=en](https://careers.ibm.com/ShowJob/Id/717577/Entry-Level-DevOps-Developer-Engineer/?lang=en)

Now, I have no CS degree or prior experience, but I'm learning a lot on my own, have a reference inside IBM, and am **thirsty** for a position in this field and will do the legwork to get there.

To you, the experienced members of r/devops, what would you say are good things a candidate for this position should be able to do. It's a bit vague in some areas, like the "You are great at solving problems" part, but I'd like to know how to focus my study. Thank you.

https://redd.it/e8usgo
@r_devops

Jira/Atlassian documentation is insane

Please tell me im not the only one going insane trying to comprehend the atlassian documentation on jira development? I would would regard myself as a competent developer but it cant just be me finding the docs hard to read and understand, it reals like its been written by an inifinite amount of monkeys!

has anyone worked on jira before, written any react apps or extended the base code with java. We are looking to invest time and resources into building an application for our business to use daily im wondering is it worth it, the documentation is putting me off and giving me red flags, should I beware of using jira?

https://redd.it/e8qdw4
@r_devops

Best options to monitor Nginx

Just wondering what people are using to monitor/graph Nginx web servers. I have A grafana/prometheus setup already in use for other metrics so I'm learning toward using one of the Nginx prometheus exporters, I see two on their official exporters page Nginx metric library and Nginx VTS exporter (not sure what the difference between the two is yet or which is better). Then there are other options such as collectd/telegraph etc. I assume using a prometheus exporter may be the best route?

https://redd.it/e8slbe
@r_devops

Issues deploying Consul onto Kops Kubernetes cluster

I'm having some issues deploying Consul via the offical helm chart to a Kubernetes cluster created using Helm. I have a 3 master, 3 worker node (testing only prod will be larger) cluster, I use the helm chart along with the latest version

Command used for deployment

helm install consul-dev ./consul-helm --namespace=development

As you can see I am using a pretty basic setup - the defaults just put into a namespace.

​

Seeing this error when I describe server 0

Warning FailedScheduling 25s (x5 over 3m28s) default-scheduler 0/6 nodes are available: 6 node(s) had taints that the pod didn't tolerate.

​

Output of describing a worker node. Looks like there is a taint but I never put that there so not sure why it'd be there.

me@LAPTOP-LINUX:~/$ kubectl describe node ip-10-0-111-37.ec2.internal
Name: ip-10-0-111-37.ec2.internal
Roles: node
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=t3.small
beta.kubernetes.io/os=linux
failure-domain.beta.kubernetes.io/region=us-east-1
failure-domain.beta.kubernetes.io/zone=us-east-1c
kops.k8s.io/instancegroup=nodes
kubernetes.io/hostname=ip-10-0-111-37.ec2.internal
kubernetes.io/role=node
node-role.kubernetes.io/node=
Annotations: flannel.alpha.coreos.com/backend-data: {"VtepMAC":"ce:d4:a9:0c:25:79"}
flannel.alpha.coreos.com/backend-type: vxlan
flannel.alpha.coreos.com/kube-subnet-manager: true
flannel.alpha.coreos.com/public-ip: 10.0.111.37
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Tue, 10 Dec 2019 18:41:24 -0500
Taints: node.kubernetes.io/unreachable:NoExecute
node.kubernetes.io/unreachable:NoSchedule
Unschedulable: false
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure Unknown Tue, 10 Dec 2019 18:43:31 -0500 Tue, 10 Dec 2019 18:44:14 -0500 NodeStatusUnknown Kubelet stopped posting node status.
DiskPressure Unknown Tue, 10 Dec 2019 18:43:31 -0500 Tue, 10 Dec 2019 18:44:14 -0500 NodeStatusUnknown Kubelet stopped posting node status.
PIDPressure Unknown Tue, 10 Dec 2019 18:43:31 -0500 Tue, 10 Dec 2019 18:44:14 -0500 NodeStatusUnknown Kubelet stopped posting node status.
Ready Unknown Tue, 10 Dec 2019 18:43:31 -0500 Tue, 10 Dec 2019 18:44:14 -0500 NodeStatusUnknown Kubelet stopped posting node status.
OutOfDisk Unknown Tue, 10 Dec 2019 18:41:24 -0500 Tue, 10 Dec 2019 18:44:14 -0500 NodeStatusNeverUpdated Kubelet never posted node status.
Addresses:
InternalIP: 10.0.111.37
Hostname: ip-10-0-111-37.ec2.internal
InternalDNS: ip-10-0-111-37.ec2.internal
Capacity:
attachable-volumes-aws-ebs: 25
cpu: 2
ephemeral-storage: 125753328Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 2013532Ki
pods: 110
Allocatable:
attachable-volumes-aws-ebs: 25
cpu: 2
ephemeral-storage: 115894266893
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 1911132Ki
pods: 110
System Info:
Machine ID: <ID>
System UUID:

<ID>
Boot ID: <ID>
Kernel Version: 4.9.0-11-amd64
OS Image: Debian GNU/Linux 9 (stretch)
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://18.6.3
Kubelet Version: v1.13.12
Kube-Proxy Version: v1.13.12
PodCIDR: 100.96.10.0/24
ProviderID: aws:///us-east-1c/i-0fd2d3538b39b1952
Non-terminated Pods: (2 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system kube-flannel-ds-fwhgt 100m (5%) 0 (0%) 100Mi (5%) 100Mi (5%) 32m
kube-system kube-proxy-ip-10-0-111-37.ec2.internal 100m (5%) 0 (0%) 0 (0%) 0 (0%) 32m
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 200m (10%) 0 (0%)
memory 100Mi (5%) 100Mi (5%)
ephemeral-storage 0 (0%) 0 (0%)
attachable-volumes-aws-ebs 0 0
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Starting 32m kubelet, ip-10-0-111-37.ec2.internal Starting kubelet.
Normal NodeHasSufficientMemory 32m (x2 over 32m) kubelet, ip-10-0-111-37.ec2.internal Node ip-10-0-111-37.ec2.internal status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 32m (x2 over 32m) kubelet, ip-10-0-111-37.ec2.internal Node ip-10-0-111-37.ec2.internal status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 32m (x2 over 32m) kubelet, ip-10-0-111-37.ec2.internal Node ip-10-0-111-37.ec2.internal status is now: NodeHasSufficientPID
Normal NodeAllocatableEnforced 32m kubelet, ip-10-0-111-37.ec2.internal Updated Node Allocatable limit across pods
Normal Starting 32m kube-proxy, ip-10-0-111-37.ec2.internal Starting kube-proxy.
Normal NodeReady 31m kubelet, ip-10-0-111-37.ec2.internal Node ip-10-0-111-37.ec2.internal status is now: NodeReady

Line matching for a node in `kubectl get nodes`

ip-10-0-111-37.ec2.internal NotReady node 36m v1.13.12

https://redd.it/e8zaeu
@r_devops

Verification of database changes?

Hey r/devops,

First off, I apologize if I am on the wrong subreddit for this, I'm a "junior devops" in my company so I felt this question would be appropriate here. I also apologize for any formatting issue or vagueness of my question.

I was wondering if I might be able to get some guidance on a problem I have.

We have a Hortonworks Hive database. I'm one of the people responsible to migrating/applying database changes using scripts that the developers have given me.

They've been asking for some automated way of tracking database changes and outputting a report of what has changed after a migration.

i.e.
Table A = col 1, 2, 3

Table A after migration = col 1, 2, 3, 4

Report = Table A after migration = col 4 added. Script that applied change to Table A successfully ran.

The program has to read the script it's about to apply, apply it, then confirm whether the change successfully applied or not. They don't want a report that they have to read through, they want it to just confirm that everything worked automatically.

The real problems are these:

1. I have ideas on how to do this but they seem extremely tedious and I don't have time to do it on top of all the other stuff they're asking me to do.

2. I basically have no tools or the ability to install tools. All I have is some extremely limited jenkins and whatever bash/python comes installed with an Amazon EC2 instance Linux Redhat OS. I can't install anything else, even python libraries. (At least server wide, if it fits in my home directory, it can work.)

Is there some application of bash/python script that can help me with this? How have others done it without anything but bash and python?

I've already looked at tools like Liquibase, but my company refuses to install anything.

Sorry for the wall of text.

https://redd.it/e909tu
@r_devops

Using dynamic inventory for better Ansible provisioning with Packer and WinRM

Hey y'all. Just wanted to share this blog I wrote up. When trying to solve this issue I found some discussion on the topic here a few months ago. Figured I'd do a new post instead of digging up an old thread.

https://medium.com/@borncamp/using-dynamic-inventory-for-better-ansible-provisioning-with-packer-and-winrm-72cd393cc19f?

https://redd.it/e8vp79
@r_devops

What sysadmin skills do I need to become better devops

I started as a back end engineer and gradually transitioned into DevOps.. I feel I need better understanding of memory allocation, CPU usage and resources.

Any sysadmin here doing DevOps? Kindly advice.

https://redd.it/e8xq7x
@r_devops

Which is better: Logz.io | Elastic.co | Loggly for logging needs?

I need to store logs from my application - roughly around 5GB/day with 15 day retention. Which of the following is best suited - [logz.io](https://logz.io/), [elastic.co](https://elastic.co/) managed service or vendors like loggly, sumologic, etc? What are the pros and cons for each?

https://redd.it/e8twrh
@r_devops

Help me Dockerify this mess!

So when developing locally i use a docker-compose file that has the following services:

\- Node.js worker (performs cron jobs by polling endpoints with a small api exposed to manually kick off processes)

\- Hasura GraphQL Server

\- React admin panel - kick off crons manually and review data

\- Postgres (uses dbaas in prod, though)

\- Redis (uses dbaas in prod, though)

&#x200B;

However, I've read that using docker-compose in production is not great practice but because this is a side-project and has but a few hundred users k8s feels like overkill. I would also like to figure out a way to use Prometheus, Grafana, ELK, and either Traefik/Caddy but throwing all of this into a docker-compose file seems very wrong... Would anyone give me some advice or steer me in the right direction? How would you do it?

https://redd.it/e8ynme
@r_devops

Environment Tracing and Build Annotations with Azure Pipelines

Neil Peterson does a great job in the following article, [Environment Tracing and Build Annotations with Azure Pipelines](https://techcommunity.microsoft.com/t5/ITOps-Talk-Blog/Environment-Tracing-and-Build-Annotations-with-Azure-Pipelines/ba-p/1056276?WT.mc_id=ITOPSTALK-reddit-abartolo), in detailing how to achieve environment traceability when using Azure Pipelines. An Azure Pipeline is essentially a series of automated tasks. Many tasks are built into Azure Pipelines or are available to install from the Visual Studio Marketplace. Some Azure Pipeline tasks include traceability annotations out of the box. The Kubernetes Manifest task deploys a Kubernetes manifest file to a Kubernetes cluster. When doing so, the objects created by the manifest file are automatically annotated with information about the Azure Pipeline build.

https://redd.it/e95jtr
@r_devops

The best courses to get to industry-recognized qualifications, skills

Hi

Cards on table: I'm going to make pitch to bring in some new technical courses to a training provider, in areas where the demand for technical skills is greater than supply, where employers are crying out for people familiar with certain technologies.

So I'd like to ask this community:

1. What technical skill or skills is a hot area right now?
2. Do those skills have a recognized qualification or certification path?
3. What would be the pre-requisites to that skill being acquired?
4. What is the best course or courses which would lead to acquiring those skills?

So for example:

1. AWS SysOps
2. AWS Certified SysOPs
3. Linux skills in administration container technologies
4. Amazon's own courses?

Thanks

https://redd.it/e8xl4v
@r_devops

Software signing solution - key storage

Hey guys,
In our company someone ages ago created some "system" to sign software we publish where the private key to the certificate is stored on HW HSM and that is connected to some "USB over IP" little box so it can connect to VM.

We're now in process of redesigning the whole signing process as it currently requires too much manual work and I'm not really a big fan of this box thingy. Naturally, we do not want to plug the HSM just into one of our servers, because what to do in case of server maintenance, right?

So I wonder how others deal with this, where do you store the private key for signing certificate ?
I'm even considering using something like Azure Key Vault - does anyone have any experience with that?

https://redd.it/e974jo
@r_devops

grok pattern

What is the grok pattern to parse data from this Log?

https://i.imgur.com/FT1P489.png

see these values are preceded by some text and an equal.

What pattern can parse the data value from such log line ?

https://redd.it/e98kz2
@r_devops

How do you handle multiple environments for your tools?

Currently, I have several scenarios where I deploy a series of auxiliary tools (an instance of Gitlab, Vault, Jenkins, Nexus, etc.) in a particular subnet for a team, separate from the subnets used for the apps they are deploying. Their apps are usually isolated on a per-environment basis, but the tools aren't, I just deploy a single instance.

This means that when, for example, we upgrade Jenkins or add a plugin to it, we are running into a risk of causing a downtime or, worse, having data loss. Should all changes to the tooling part of the infrastructures be tested in separate environments, like we do with our own apps? Is there any best practice for this?

https://redd.it/e95riv
@r_devops