My DevOps Learnings in 2019. List Yours:)

1. DevOps is not an individual role, it is a group effort.
2. Your infra should be able to handle a failed deployment automatically and rollback. If it can’t, you’re doing it wrong.
3. DevOps means different things to different people.
4. If any task cannot be automated, remove it instantly from the task list.
5. Communication is more important than tooling and will make or break an organization.
6. Shift the focus of Security to the left in the development life-cycle.
7. Jenkins != CI/CD …
8. Always design with autoscaling in mind. Automate-all-the-things. One-click deploy to one-click revert.
9. Create ready to use images in AWS and use those images for auto-scaling instead of scripted from GitHub builds.
10. There is a real skillset shortage in the DevOps field.
11. The CI/CD pipeline is one of the best practices for DevOps teams to implement, for delivering code changes more frequently and reliably.
12. Do not automate a bad process, you end up with an automated bad process — \[Credits: DOES18 \]
13. Testers who know how to code and automate scripts to test various cases are in massive demand in DevOps.
14. Add monitoring to infrastructure as well as applications. The monitoring system should send alerts on every important events.
15. Utilize Infrastructure as a code templates to build the infra. Ex. cloud-formation, terraform etc
16. If you need to SSH into your VM then your automation is failed.
17. Serverless is not actually serverless, it is simply 'pay as you use' model
18. Cloud-native is not a synonym for microservices or Kubernetes, it is all about utilizing the advantages of the cloud computing model

https://redd.it/eg6jjf
@r_devops

Terraform single vs multiple workspace for team collaboration

Hi guys

We are migrating to new cloud vendor and starting to use terraform to build our infrastructure with code.

We have many people in our team, and probably have around 5 people who will be focused on developing terraforms. Within these 5, only 2 will have super admin privilege (full access in cloud stacks)

We are making use of terraform modules, where each module is dedicated for specific needs (i.e. network, instance, etc)

While i’m still learning beat practices with terraform, i see the following 2 options at this point (please note i’m a beginner)

1) One master terraform workspace where each module is a subfolder. Terraform state will be stored in cloud object storage (like s3), and multiple people will use this single workspace for developement. Pro is we have only one place where we define our infrastructure, and our modules are in centeal workspace. Con is that it makes it difficult to control the blast radius - theoretically, non admin who does not have full infra knowledge can make mistakes or push incorrect changes that can screw up something like network infrastructure

2) Multiple terraform workspaces.
For example, one workspace will be dedicated for network (vpn, subnets, routes, etc) where only admins have access to develop. In our shop, network is something that does not change very often.
The other workspace will be dedicated for something like compute services (i.e. provisioning new ec2 instance) - everyone will have access to this. Even if someone screws up something, the worst they can so is destroy instances or whatever the scope of that workspace is.
Downside here is now our infrastructure is defined in multiple places, modules will be scattered, will create some dependencies (i.e. network needs to be set up before you can provision instance in the right network)


For me option #2 feels more appropriate since it will make it easy to divide the work between full admin and non admin. It will also reduce blast radius in case something goes wrong.

What do you guys think? Maybe I’m not understanding terraform very well therefore not making an educated assessment. If you guys are using terraform in your shop, please share how you manage your workspaces and collaborate within your team.

https://redd.it/eg53rs
@r_devops

What do you use for infrastructure CD

Hey there. I'm just wondering what everyone is using for automatically updating infrastructure. I'm currently trying to get all my provisioning and configuration as code, and while it works great manually, pipelining it all together seems a bit of a task, especially with lots of inputs and outputs of different tasks. We currently have a multiple Jenkins pipelines, with the "main" one that provisions a k8s cluster looking like the followings:
- Terraform to provision VMS. This gives us a tfstate file, which also gets outputted to Jenkins as an artifact.
- Ansible to configure each VM, using Inventory from our tfstate, using the Terraform inventory plugin. This runs multiple playbooks but not all, because we need to...
- Bootstrap Rancher via terraform, which is running as a lone container. This will give us a Rancher URL and token, which can be used to...
- Create a K8S RKE cluster via Terraform. This will give us a cluster URL and a Token which can be used to...
- Install Rancher Agent on each VM via Ansible

As you can see, lots of input is getting piped back and forth, with no easy way of seeing what is and isn't working, without looking at the text output from each job. Does this seem right to you guys? Ideally we would have one big playbook which would do everything, but Ansible and Terraform both have different strengths at different things. All the current CICD tooling seems to be really dev focused

https://redd.it/eg155y
@r_devops

Associate Devops role or stay as a sys admin?

Hi guys,

​

Recently been offered a role to an associate devops for a msp. It is what I have been looking for, except, its a junior role and the pay is slightly lower than mine (8k aud less, I am on 98k)

I am a sr sys admin, but my work dont have much to do with clouds and they lack the cloud vision to moving forward as a company. The offered company has been cloud Consultant in aus for some time now, so they have some big clients like major banks and Airlines.

What would you guys do? Stay as a sys admin / potentially becoming a SE or go to a junior devops role and move on from there?

https://redd.it/eg42m5
@r_devops

elassandra questions

Hey guys,

My company and I are looking into setting up elassandra to be one of our data stores. We want to start small, so what would the difficulty be in setting up a single elassandra cluster? Also, are there any good resources out there besides the docs?

https://redd.it/efz9ek
@r_devops

write ansible playbook that outputs each configuration items in a format that can be imported into a spreadsheet in tab-delimited format

I need to write a playbook an output printing in csv file using python. The output is the ip address config.

https://redd.it/efykws
@r_devops

What’s in store for Kubernetes Operators in 2020?

Here are some thoughts on how Kubernetes Operator movement will evolve in the coming year.

[https://medium.com/@cloudark/whats-in-store-for-kubernetes-operators-in-2020-98263aa23fb9](https://medium.com/@cloudark/whats-in-store-for-kubernetes-operators-in-2020-98263aa23fb9)

Will be happy to discuss any thoughts or questions that you may have.

https://redd.it/egb7as
@r_devops

Using Graylog for doing cool dashboards

Hey buds! I'm kind of newbie in the devops world, so now I'm using Graylog in order to know what it is happening with the infraestructure.

I'm trying to create some histograms with the response codes of my server, but Graylog show me all the messages since 1970! I read that I should add some boundaries but I dont know how to do it. Does reddit can help me? 😁😁

https://redd.it/efy0uw
@r_devops

write ansible playbook that outputs each configuration items in a format that can be imported into a spreadsheet in tab-delimited format

I am trying to write an playbook to parse the output in a csv file using python. Each output is an ip address config. i need the output to be send a csv file. I tried youtube videos but it does not help me.

https://redd.it/efxrfb
@r_devops

Pay to play

A company in Germany asked me to solve a coding problem during a job interview process, but they want me to host a complete API at my own cost at GCP.

Writing a program to demonstrate I know what I am doing is one thing, but it seems they ask a little bit too much. Especially the part where I need to pay money just to apply to a job.

My first thought was "Have they lost their marbles?".

What do you think?

https://redd.it/efv7ns
@r_devops

Migrate large size or Mongodb and MYSQL to another server (low bandwidth)

Hey guys,

Any tips about migrating/cloning large size of mongodb databse and MySQL (100gb++) from one server to another.
Also, I’m on AWS and their instance are on China cloud services, so the network connection could be unstable.

https://redd.it/efwtwl
@r_devops

Hashicorp Vault for temporary aws/gcp/az web console logins?

Hey
I work for a team that often has to give Cloud related demos, workshops, etc.. at schools or other organizations. The demo accounts we use for this are usually created manually now, but we'd like to provision these with an expiration date automatically so we don't have to think about credentials being left active.
These are fairly low level workshops so all we need is an ability to generate a few accounts with Console access and limited permissions to spin up a VM etc. On all 3 major cloud platforms.

Hashicorp Vault seemed to be a great platform-independent tool at first glance but it looks like it is more aimed at provisioning service accounts.
Can this be done Vault? Is there a better tool for our use-case?
Thanks

https://redd.it/efvk7s
@r_devops

Single dev startup: Moving from Heroku to ???

The company I work for runs everything in Heroku. It does a good job and allows us to focus on building stuff, but now that the user base is growing so are the costs... So we are considering other options.

For context, our backend consists of a pretty standard Django/DRF monolith plus Celery, backed by Postgres and Redis. I am the sole backend dev and have very ilttle devops experience, so I want to avoid something overly complex or that will use up too much of my time. I am comfortable with Docker, but not with Kubernetes/swarm/etc.

A self-hosted Heroku would be the holy grail I guess, but I don't think it exists. I've used Dokku before and it works great, but doesn't provide clustering/HA. Flynn is similar and supports clustering, but the docs aren't great and the database support is described as "experimental".

So I've gone on to think about more complex alternatives. Maybe a cluster with docker swarm? Maybe managed Kubernetes from GCP or DO? Maybe Beanstalk or similar?

Has anyone made a similar transition or have any ideas on how to do this without embarking on a massive learning journey/taking up all my time?

https://redd.it/efsjo0
@r_devops

OUR SUBREDDIT MADE IT TO THIS BLOG

[https://www.portshift.io/blog/kubernetes-cloud-resources-devops/](https://www.portshift.io/blog/kubernetes-cloud-resources-devops/)

We got a shoutout!

https://redd.it/efuve4
@r_devops

Internal-only TLS certs from Let's Encrypt

I found a project that automates TLS certificates from Let's Encrypt that looked really great! Wanted to share


https://github.com/bradfitz/autocertdelegate

https://redd.it/efkvrw
@r_devops

Job offer to associate devops

Hi guys,

Recently been offered a role to an associate devops role for a msp. It is what I have been looking for, except, its a junior role and the pay is slightly lower than mine (8k aud less, I am on 98k)

I am a sr sys admin, but my work dont have much to do with clouds and they lack the cloud vision to move forward as a cloud company. The offered company has been cloud Consultant in aus for some time now, so big clients like major banks and Airlines.

What would you guys do? Stay with potentially becoming a SE or go to a junior devops role?

https://redd.it/efmq5w
@r_devops

Docker consumes too much of storage space

Hi everyone,
I have been working in DevOps from the past 2 years now. Everyday we face challenges that we somehow manage to resolve, but there are some challenges that still needs to be addressed by me and I think I have researched a lot and haven't got any relevant solution.

I have an Ubuntu EC2 instance on which I do PoC's using Docker. In order to create a container I build a docker image from scratch and then I run that docker image. When I'm done with my PoC I usually remove that container and it's associated image, then during night my cron job runs a script that triggers docker system prune in order to remove dangling images/containers/volumes.

I have observed that docker still is filling up my storage space even after deletion of all the images/containers/volumes. I have also cleared the system cache, but somehow storage space is filling up fast.

Any idea/suggestions on what I'm missing or what the issue is?


Thanks

https://redd.it/efgfnu
@r_devops

Ansible Vulnerabilities 2019

Hey,

Christmas time and ansible does not get unnoticed throw the security team

Read all about it:

[https://blog.firosolutions.com/exploits/ansible-monday-2019/](https://blog.firosolutions.com/exploits/ansible-monday-2019/)

https://redd.it/efftnb
@r_devops

Continuous Infrastructure Testing, how much do you all do it today?

Obviously testing for applications is a no brainer and best practice....there are numerous approaches including TDD for traditional software engineering. But for infrastructure as code I feel like testing gets discussed far less often. To what extent are people testing their infrastructure code? What tools and frameworks if any do you use? What pain points have you run into?

https://redd.it/efdzzl
@r_devops

Upcoming Project - Is Kubernetes the right choice?

I am currently tasked with building some automation around enduser's requesting the provisioning of a long-lived Amazon Linux/Windows Servers in our AWS environment.

These servers will host a wide range of services for customers e.g. file storage, database server, etc. This service is primarily a "We give you the Infrastructure. You do what you want with it".

Would it beneficial to develop the infrastructure around Linux/Windows containers in a Kubernetes cluster via AWS EKS?

Or would it make more sense to spin up long lived EC2 instances and manage them that way?

Thanks!

https://redd.it/efdf0w
@r_devops