grok pattern

What is the grok pattern to parse data from this Log?

https://i.imgur.com/FT1P489.png

see these values are preceded by some text and an equal.

What pattern can parse the data value from such log line ?

https://redd.it/e98kz2
@r_devops

How do you handle multiple environments for your tools?

Currently, I have several scenarios where I deploy a series of auxiliary tools (an instance of Gitlab, Vault, Jenkins, Nexus, etc.) in a particular subnet for a team, separate from the subnets used for the apps they are deploying. Their apps are usually isolated on a per-environment basis, but the tools aren't, I just deploy a single instance.

This means that when, for example, we upgrade Jenkins or add a plugin to it, we are running into a risk of causing a downtime or, worse, having data loss. Should all changes to the tooling part of the infrastructures be tested in separate environments, like we do with our own apps? Is there any best practice for this?

https://redd.it/e95riv
@r_devops

Lost in the Security Rabbit Hole with a side of Logging

Hello r/devops,

I am very new to this space but have been loving it so far and seeing how helpful this community is, it motivated me towards leaving my trusty shadow cave and come out of lurking.

I and my team inadvertently found ourselves having to do a ton of DevOps implementations due to the recent pivot of our company. We are a team of 4 developers with next to none operations or sysadmin experience. We've faired well until now, due to limited requirements from us. The silver lining is that we are given complete control of the DevOps infrastructure project, which is being built from scratch) and we are learning, experimenting, and implementing slowly but surely. However, we found ourselves in a bit of a rabbit hole on two subjects: Security and Logging.

We would like to get your opinions and pointers on potential solutions to these two components. A bit of background distilled into pointers:

* The infrastructure will host a financial services platform that will be used by financial institutions as well as individual investors, academics and hobbyists (Though in a limited manner than institutions).
* Domain oriented micro service architecture.
* A service will contain its required components in itself such as caching and database.
* AWS is the main service provider.
* EC2, CloudWatch, Lambda, S3, and a few others
* The main programming language is Java with heavy usage of Spring Framework.
* Sensitive information will flow in and out of the system.
* Client communications and all communications between individual components must be secured.
* We have to be able to utilize directories of our customers as a means of authentication and authorization.
* We are required to record everything (some sensitive information as well) in logs and store these logs for certain periods based on metrics defined by law. Such as:
* Logs that are less than 12 months old need to be accessible within the hour.
* Logs older than 5 years must be accessible within the month.
* Everything must be stored for 20 years (Crazy).
* A small team with a lot of responsibility means we have to go for fewer cogs to achieve the desired result. We would love to find the best solution for our problems but we prefer having a single component to achieve 80-90% rather than 8 components to achieve 100%.

**Where we are at:**

Logging and Security are not the only two aspects we haven't figured out yet but they are the next on our project plan. I will most likely ask about other components on this subreddit when we get there. In terms of these two components, our progress so far:

* We are stuck on ELK vs Graylog. ELK seems to be the go-to choice at the moment but Graylog seems easier to use and has fewer moving parts. It is safe to say we are leaning towards Graylog.
* We identified a couple of Open Source projects that offer a plethora of security-related features and we distilled the list into two options: Gluu and KeyCloak. None of us has a deep understanding of this subject as our requirements for security implementations have been very limited so far. We imagine that we will have to utilise some AWS features along with these but there is just too much stuff! This area seems like a proper rabbit hole that we end up more confused than before. Our identified requirements and thought solutions are as follows:
* Microservices among themselves. Also, microservices and their databases.
* Behind private subnets and firewalls
* No internet access
* Services and API Gateway (Haven't decided on the gateway yet).
* Public gateway accesses Public Load Balancer
* Public LB forwards to firewall and intrusion detection and prevention (ID/P) layer
* If all is well, firewall - ID/P layer forwards request to private LB
* Private LB authenticates the request via Gluu/Keycloak (That resides on a separate environment)
* Customers and API Gateway
* Gluu/Keycloak and Gateway features?
* Gluu/Keycloak
* Behind a public LB and firewall - ID/P laye

r?
* Access to Customer LDAP (or similar)
* Private VPN between Gluu/Keycloak and customer system?

**What you can help with:**

Of course, if you have any comments, questions, pointers or answers to anything you see here, you are of course more than welcome to do so. However, our list of unanswered questions are as follows:

1. We couldn't find a solution to fit our log archival requirements and we decided on custom building it into our job automation system. Did we miss something or is this actually the case?
2. Is our understanding of Graylog vs ELK accurate? Unless we grossly overlooked a very important aspect in our comparison of the two, we will be going with Graylog. What is your opinion on this?
3. How would you go about implementing a security solution for a system like this? Does this seem like overkill or is it appropriate?

Thank you for taking the time and to read this far!

https://redd.it/e96365
@r_devops

Any opinions on fastly vs cloudflare?

A bit of context: mid-size startup, currently evaluating CDNs that can support some fancy logic on the edge. Lambda@Edge seems logical given that we're mostly on AWS, but after just paging through its documentation I had a distinct impression of a clunky, half-baked product. This leaves Cloudflare (with their Workers) and Fastly (they have Webassembly on the edge in their labs). We push a few TB per month out, ever increasing, so we're probably looking at Enterprise pricing from Cloudflare and it's in low $1000s/mo, which is OK. However, we didn't have any experience with neither. From my perspective, cloudflare seems a bit more hip and trendy; fastly is apparently older and less likely to get bought out or suddenly change. What do you think folks? What are the pros and cons of going with one or another?

https://redd.it/e9cbms
@r_devops

What does a DevOps (practitioner/engineer) do during a "Normal" day of work?

Probably an incredibly silly question but I am just trying to envision and understand what the actual job looks like....

What tools do DevOps folk use? Any IDEs?

What problems are they solving on a daily basis?

Do they code? If so what languages? And what are they coding?

Just trying to get a deep thorough understanding of what the job itself really looks like in terms of tasks, actions & daily chores? I would really appreciate any DevOps folk to describe any parts of their day to day work?

Also sneaky extra question... What is the difference between a DevOps practitioner & a DevOps engineer?

Thanks for your help

https://redd.it/e94huc
@r_devops

Cant join worker nodes in Kubernetes on V1.17

Im using a cluster I created on 1.16.3 and worker nodes are stuck at NotReady... I created additional nodes but they never show to Ready state..it looks like they are on 1.17 I dont know if its some compatability issue

​

kubectl describe node shows :

Ready False Wed, 11 Dec 2019 07:44:31 +0000 Wed, 11 Dec 2019 07:35:27 +0000 KubeletNotReady runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

​

I tried upgrading the master control plane to 1.17 and one node joins but the other gets stuck. IT looks like I only have two worker nodes max Im trying to have 3.

https://redd.it/e9446v
@r_devops

What's recommended for homebrew tek into employment?

I'm getting started on my DevOps homebrew teks soon. It has taken me awhile to get into it. I have the book by Sobell "A Practical Guide to Linux" which is also my college textbook. I'm taking "Advanced Linux" and "Intermediate Python Programming" at local college to aid my resume and have skill. Ok so I have LinuxAcademy am thinking I will need Git, Docker, Kubernetes, Jenkins, Ansible, etc. Pretty cool I have everything I need also I put in my i5 quad core laptop 16GB of RAM so I can actually simulate things like spinning up instances either locally or on the cloud I want to embrace AWS I have my "free tier" account used it for the Python development course on LinuxAcademy. Basically if there is anything recommended for employment I'd take advice. My attempts at employment lead to a lot of rejection finally after questioning reality I've decided to make strides at skill and having talent would serve me well. Like for example, I could come off like I enjoy the work ask more questions, and show the employer what I can do hands on with the laptop at their office. I'm casual about things but want to make sure my path leads to a bulls eye mark of hire.

Python and GO are on the to do list. Ruby might be useful for Vagrant and such but idk if I can master 3 languages. I want to do artifacts of SCMs like Git and be really sophisticated at this very devoted and able to make a good paycheck. Thanks! Want to do Terraform.. Have heard of Perforce and other tools too. Basically if there are a list of tools that do artifacts and else I should be learning/be expert at please let me know here..Like what all do I need of Git... I can do checkouts and merges. I did a cherry picking exercise with a simulator as well from a Stanford friend. Thank you!

https://redd.it/e92l3z
@r_devops

How does one actually microservice?

This feels like a weird question in almost-2020 but it’s serious.

You can find countless videos and blog posts on why to do the micro services and what high level architecture looks like.

There’s lots of discussions around integration technologies, synchronous vs asynchronous communication, CQRS, circuit breaking, service discovery.

What I can’t find is how people _actually_ implement these architectures in the real world.

How do people handle different stages?

Do services (including the client applications) just always consume the prod version of other services?

Do you build a giant service-stage matrix to keep track of what stage of a service you communicate with?

How are you handling data that needs shared across different services?

What do you do when a new service needs historical data from an older service?

What do you do when inevitably build too granular of services and need to merge some together?

How to handle multiple auth needs, such application to service versus service to service?


I just have so many seemingly basic questions that I can’t find good answers to and I’m losing faith that people actually implement these kinds of system in the real world. Does anybody have some good pointers for me here?

https://redd.it/e9gz44
@r_devops

DevOps lessons from the ski industry

Since devops has learned lessons from a lot of other industries (nasa, etc), thought that this post about lessons from the ski industry would be of interest:

[https://medium.com/@nedmcclain/devops-lessons-from-the-ski-industry-e90cd4ae3633](https://medium.com/@nedmcclain/devops-lessons-from-the-ski-industry-e90cd4ae3633)

https://redd.it/e9c1y1
@r_devops

Burnt out, I think

So this is another “burnt out maybe; not sure what to do about it.”

I’ve been in the IT field since 2009 or so; I started out in the data center, then went to AWS support in Amazon, finally left AWS to be an actual devops engineer. I have been in several small companies, average one year per job since 2015.

I do like doing dev work just not so much ops work. For example, I created tools that would manage over 10k route53 records (this was for a n anti phishing company before terraform was a thing). I recently got my hands dirty with golang in creating a file stream from ftp to s3.

I just don’t care for the Ops work, I get that there are very fancy new cool tools like kubernetes which make life a little easier to manage at scale but I can’t say that work is for me.

My dilemma is this; if I go for dev work, with very little actual dev work experience, I probably won’t get hired elsewhere. Also I may not be looking to leave my company yet. Yes I’m bored but not I’ve been here less than a year and I may want to stick it out a bit longer. Then there’s the roles of director/manager, I’m not certain if I want to go that route yet.

I guess my final question is this. Is anyone else in my position, if so and you have move passed it, how did you do it?

Also, for those who say “go do some personal projects”, I have a few Alexa skills that I wrote in python and go. They are unreleased because I only use them for my device and don’t want to manage customer data.

https://redd.it/e9fxbt
@r_devops

Suggestions for a Learning Platform?

My employer is kindly paying for me to join an e-learning platform.

What’s an e-learning platform that’s really excellent? Right now it’s between Linux Academy and ACloudGuru, but I’m open to other suggestions.

https://redd.it/e8vafc
@r_devops

OpenLegacy (or alt) review ?

Hi guys,

Recently, some of the managers I work for went crazy because they heard about OpenLegacy, a solution that can "easily and automatically create open APIs from your main (legacy) systems, such as AS / 400, iSeries, etc. "

I do not really understand how we can trust something like this?

When I check each demo, the use cases are so simple and do not take into account more than 30 years of huge Cobol programs, multiple databases and half-developed Web services.

Has anyone here ever experienced this kind of solution? I am not looking for a sales pitch, which is why I am interested in what you think about it.

https://redd.it/e8xjqk
@r_devops

Amazon Simple Email Service (SES) Are Cron Jobs all you need to manage the rate of newsletter email issuing?

Good day, everyone.

I'm putting together automated notifications about new content on our website and we are looking to automate the process. I have a database with the table of new content and a table of subscribers and I essentially need to send them all an email when new content is uploaded. From what I understand, there are limits to how many emails you can send out using Amazons SES per second. How do you handle email sending? Do you Simpy set up a Cron job to trigger a PHP page every minute and handle sending emails using something like phpmailer?.

https://redd.it/e8tuvi
@r_devops

Can you give advice/feedback on part 2 of our postmortems piece?

Hi guys! Steve McGhee worked with us on creating a second part to our postmortems article. It would be awesome if you took a look and maybe commented on what you liked and found helpful vs what you would like explained more in the future.

[https://www.blameless.com/building-reliability-through-culture-sre-steve-mcghee/](https://www.blameless.com/building-reliability-through-culture-sre-steve-mcghee/)

https://redd.it/e8ts6c
@r_devops

Why run Jenkins master in a Kubernetes cluster?

Hi,

TL;DR: kubernetes novice here, don't see why should I have the master in a cluster.

I'm looking into migrating an existing CI infrastructure (just CI, no CD), which is a fairly classic AWS EC2 based infrastructure, into Google Cloud Kubernetes Engine.

Right now Jenkins master is running on a VM in AWS, slaves are managed by the EC2 Cloud plugin, provisioned with Chef, custom AMIs built with packer, terraform etc. Each node has docker installed and Jenkins pipelines are using the docker plugin to build and execute stuff for different projects inside the containers.

It makes perfect sense for me to have the agents as containers in a Kubernetes cluster. It makes perfect sense to containerise my Jenkins master and run it as docker container, for easier upgrades, etc.

However I cannot understand why it is better to have the Jenkins master inside the Kubernetes cluster as well, it seems an unnecessary complication. I'm finding a lot of documentation on how to run a Kubernetes cluster with Jenkins, but none seems to explain why this is a good idea.

Consider the following picture:

[https://imgur.com/a/0GW8qW1](https://imgur.com/a/0GW8qW1)

vs having the jenkins master container as part of the kubernetes cluster itself.

I'm not planning on having multiple masters or do any fancy load balancing. I still can't find a good reason why everyone seems to run the master as part of kubernetes. I know there are helm apps which make the deploy of it a click install, but to me it seems I'm losing some flexibility here. Having only the agents in the cluster makes sense, since I can destroy and recreate the cluster. Maybe having 2 clusters is a better idea - one for master and one for the agents.

Anyway, sorry for the long post, but I'd really like to understand what are advantages/disadvantages of having the jenkins master container in a kubernetes cluster vs running it as a container in a classic VM.

Thanks!

https://redd.it/e8scng
@r_devops

Adding Serverspec test to Jenkins pipeline

Question from a newbie in the area.

I'm trying to add a Serverspec step in a Jenkins pipeline. The aim is to use the automated test to check if a said server contains the appropriate AMI (AMI being build with Packer) I have no idea where to start with this, and I mean from the very start, i.e: where does it go in Jenkins, folder structure, how the testing could work. Has anyone done this before or have a good reference for it.

https://redd.it/e8pqgi
@r_devops

Code structure to build a single image and apply multiple environments.

Hi, my source code and manifest are stored in the same repository.

Every time the developer commits new code, CI bake and deploys an image to Kubernetes. Target Namespace and Cluster selected based on the source code's branch. ( e.g. branch test/env1-> deploy to namespace env1 @ testbed ).

This strategy is easy and simple but can't archive the immutable infrastructure.. since CI builds an image for development and production separately.

I thinking about having a single image apply to all environments. To archive that I think I need to separate code from the manifest.

That will make the code structure make complicated. So I would like to listen to your advice.

Thank you.

https://redd.it/e8oios
@r_devops

First DevOps project -> Start with using AWS?

TLDR:

Team asked me to set up something that can serve as a dev environment and showcase current status of projects to simplify collaborating with clients. Send halp! Team leaning towards using AWS.

​

Full:

I usually do front-end dev work and am currently working for a consultancy company. Company is cool but some things are very old school and take a lot of meetings to change. One thing my team wants is to have some sort of portfolio-like environment to hold our current projects where we can work locally, push up our changes, and have some sort of pipeline that updates this portfolio so that we can more easily see the most current version of whatever we're working on as well as have an easy way to have our clients see where we are at and get feedback at all stages of development. My team figures it will be quicker to just build our own and, since none of us are experts, just get into AWS because we're gluttons for punishment, it is industry standard, they have a million tools and products, and it is something we could keep building on to try new ideas in the long term.

I have no idea what I'm doing but I'm happy to learn whatever I need to get this done, I just need:

1.) Feedback if AWS is the right choice for us or did we just pick that because we're too clueless to know that there is something better for the long-haul.

2.) What products on AWS I should be focusing on to get this done. So far I think just a simple S3 account to hold static projects and we could have some sort of login that can be created for clients to only see their project. I don't know if we'll have more complicated projects in the future (like ones that require server traffic) and not sure how hard it is to migrate stuff or add that kind of functionality later. I'd rather put in the extra work at the beginning to future proof this some (and because if I'm going to do this I might as well do this right) but also do not want to just waste company time and money on something that we will never use.

Any pointers would be greatly appreciated!

Thanks so much in advance :D

https://redd.it/e9pame
@r_devops

Suggestion request: a kubernetes-like service discovery without the cluster

I am hoping I can get some interesting ideas from people with more experience and knowledge in this domain. What I basically want to achieve is having multiple nodes, on each node having multiple services names a, b, c in any combination (each node can have any combination of a, b, c instances). I want each service to be able to communicate with other services via dns with the service names, so when a makes a request to [https://b/api/resource](https://b/api/resource), it will go to a container with the b service on any node. I am thinking that some kind of mesh proxy sitting locally that will route the traffic as needed would be the best to go, but I can't seem to find anything that can run outside of a kubernetes cluster.

I looked into envoy proxy but it seems like a more cloud native nginx alternative and other solutions seem to target specifically the kubernetes market. Having a kubernetes cluster is not a possibility for me as it has quite some overhead with the control plane and is too much overkill just for what I am planning to do (probably 3 nodes with some replicas).

Another solution I thought about is having a nginx in front on each node that knows about the local services and can route based on host, a "centralized" load balancer where all service name dns will point to and when a services makes a request to [https://b/api/resource](https://b/api/resource), it will be routed by the main load balancer to a nginx that has at least a container with **b** running

What other (open source) solutions are there that be more appropriate for these kinds of requirements?

https://redd.it/e9npsg
@r_devops