Which CI/CD mechanism do you all use?

I was just curious which mechanism did you guys find efficient among the all you used till now.
I know everyone has a different way of doing it, which one did you like.

Thanks.

https://redd.it/efqoz6
@r_devops

Automating Developer Workspaces

Hello,

I'm trying to provision a fair amount of developer workspaces, in all 3 of the major OSes, with a few different types of preset application packs based on the role of the user.

Currently, my strategy is to try out Packer with Terraform or Ansible and try to automatically install these fresh images on empty machines, but I'm not sure exactly if it is possible yet to achieve this with these tools.

Also, would it be possible to somehow connect some laptops to our network and get the OS images to autoinstall themselves?

Anyone from the fields that has any advice? :D Much appreciated

https://redd.it/eclyr7
@r_devops

Alternatives to DockerHub

I'm paying $22/month for DockerHub, which is pretty pricey.

I have my own k8s cluster, and I'm wondering what the best way to either create my own registry or use a cheaper alternative.

- Create a service inside of my cluster.
- Create an EC2 instance outside the cluster.
- Use a 3rd party.

Any insight?

https://redd.it/ech512
@r_devops

Artifactory system level importing and exporting from the command line?

Im trying to replicate an existing Artifactory server into an offline environment. Does anyone have experience with jfrog products? Im looking to automate the export of the systems but I cant find any docs on how to do it other than from the GUI.

https://redd.it/ecirlv
@r_devops

Implementing a CI pipeline using Perforce

Hello there,
I'm working in devops for a game Dev company and our depot size is ~ 200gb - Building the game locally takes ~ 20 mins. We are currently evaluating ways to improve/redevelop our CI pipeline.
Current setup polls the swarm server for new reviews and builds the shelves associated with them but since artists & game designers don't use swarm...it's just semi-helpful.
We have several architectures in our mind (submit trigger, staging depot) but each has some serious drawbacks.
What's the established, usual way to implement a CI pipeline with p4?

https://redd.it/ecipbr
@r_devops

Traffic generation tool that can be used both for baselines and stress tests

Hey everyone, here's one that I think is a relatively straightforward question but I haven't found a solid answer to online. We're trying to run some load testing on our sites, and ideally use one tool for both baseline measuring and stress testing. The trouble I've run into is that some tools out there are great for generating a bunch of traffic and hammering endpoints, thus giving consistent request/response time baselines (like apache bench), and some tools are great at generating a huge load of realistic user behvior (like locust). We've tried running both baseline testing and performance testing with locust, but we've found some marked unreliability in terms of request/response times with locust, indicating to me that there's either some noise in our test infrastructure or the test itself is not keeping all variables the same and is thus unsuitable for baselines.

Have any of you out there used one tool for both use-cases, or are they measured differently inherently and worth using two separate tools for?

https://redd.it/ecfzmh
@r_devops

Where can I start to modernize my DevOps environment?

Please read through my quick overview of our environments at my company, and let me know where you think is a good place to start in order to get this environment into a more modern DevOps culture.

* Production and staging’s technology stack is as follows:
- Legacy .NET Framework APIs running on 8 windows 2008 R2 IIS Servers (EC2 instances).
-Data is managed by a very large monolithic SQL server cluster (2 x i3.16xlarge and 1 i3.8xlarge).
-The new microservices are running on AWS Lambda/DynamoDB/S3 – each with their own Cloudformation templates.
* The development environments:
-20 different dev environments running a single EC2 instance each, and each holding all of the legacy APIs on that 1 EC2.
-Each dev environment also has its own EC2 running SQL server 2008
-The microservices/lambdas for the dev environments are created by me on a per-needed basis.
-Basically if a software developer is working on one of the serverless services, they will come to me to add that service to their development environment.

* Source Control and the CI/CD Pipeline:
-Microsoft’s Team Foundation Server (TFS)
-Developers check in the code to TFS and then manually queue a build from the TFS console
-Deployments are manually initiated on TFS, but then executed by AWS CodeDeploy. CodeDeploy uses the CodeDeploy agent to talk to the dev environment EC2 instances to deploy the code (which was zipped up to S3 via TFS tasks).
-The microservices are deployed via powershell scripts (AWS tools for powershell) from the TFS deploy server

Though our developers try to follow Agile principles, we very much do Waterfall deployments. We are unable to do blue/green or rolling deployments because of the massive legacy SQL server. Every deployment that I’ve run, the SQL server has had very large schema changes which require us to run migration scripts to update the schema.

The goal of the development team is to breakup the endpoints that live in the legacy .NET framework APIs and convert them over to microservices while also removing the dependency of the large SQL server. But in the meantime, every month or so we have a 6 hour outage on a Friday evening to do a large Waterfall deployment.

The things that I want to do in order to change culture here would require such a massive overhaul, I just don’t know if it’s possible without a huge investment from the company.
I guess in the meantime I would like some suggestions on anything you see fit to suggest.
Here are a couple of big-ticket problem items that I can’t figure out:

* How to better manage having a ridiculous 20 dev environments (EC2+Sql server combo) with microservices added to each as-needed.
* Assuming the SQL server dependency doesn’t change for a while- how to better manage that
* Best entry point to start engineering for live production releases rather than a Friday evening downtime once a month.
* Anything else you can think of

https://redd.it/ec3pwl
@r_devops

My DevOps Learnings in 2019. List Yours:)

1. DevOps is not an individual role, it is a group effort.
2. Your infra should be able to handle a failed deployment automatically and rollback. If it can’t, you’re doing it wrong.
3. DevOps means different things to different people.
4. If any task cannot be automated, remove it instantly from the task list.
5. Communication is more important than tooling and will make or break an organization.
6. Shift the focus of Security to the left in the development life-cycle.
7. Jenkins != CI/CD …
8. Always design with autoscaling in mind. Automate-all-the-things. One-click deploy to one-click revert.
9. Create ready to use images in AWS and use those images for auto-scaling instead of scripted from GitHub builds.
10. There is a real skillset shortage in the DevOps field.
11. The CI/CD pipeline is one of the best practices for DevOps teams to implement, for delivering code changes more frequently and reliably.
12. Do not automate a bad process, you end up with an automated bad process — \[Credits: DOES18 \]
13. Testers who know how to code and automate scripts to test various cases are in massive demand in DevOps.
14. Add monitoring to infrastructure as well as applications. The monitoring system should send alerts on every important events.
15. Utilize Infrastructure as a code templates to build the infra. Ex. cloud-formation, terraform etc
16. If you need to SSH into your VM then your automation is failed.
17. Serverless is not actually serverless, it is simply 'pay as you use' model
18. Cloud-native is not a synonym for microservices or Kubernetes, it is all about utilizing the advantages of the cloud computing model

https://redd.it/eg6jjf
@r_devops

Terraform single vs multiple workspace for team collaboration

Hi guys

We are migrating to new cloud vendor and starting to use terraform to build our infrastructure with code.

We have many people in our team, and probably have around 5 people who will be focused on developing terraforms. Within these 5, only 2 will have super admin privilege (full access in cloud stacks)

We are making use of terraform modules, where each module is dedicated for specific needs (i.e. network, instance, etc)

While i’m still learning beat practices with terraform, i see the following 2 options at this point (please note i’m a beginner)

1) One master terraform workspace where each module is a subfolder. Terraform state will be stored in cloud object storage (like s3), and multiple people will use this single workspace for developement. Pro is we have only one place where we define our infrastructure, and our modules are in centeal workspace. Con is that it makes it difficult to control the blast radius - theoretically, non admin who does not have full infra knowledge can make mistakes or push incorrect changes that can screw up something like network infrastructure

2) Multiple terraform workspaces.
For example, one workspace will be dedicated for network (vpn, subnets, routes, etc) where only admins have access to develop. In our shop, network is something that does not change very often.
The other workspace will be dedicated for something like compute services (i.e. provisioning new ec2 instance) - everyone will have access to this. Even if someone screws up something, the worst they can so is destroy instances or whatever the scope of that workspace is.
Downside here is now our infrastructure is defined in multiple places, modules will be scattered, will create some dependencies (i.e. network needs to be set up before you can provision instance in the right network)


For me option #2 feels more appropriate since it will make it easy to divide the work between full admin and non admin. It will also reduce blast radius in case something goes wrong.

What do you guys think? Maybe I’m not understanding terraform very well therefore not making an educated assessment. If you guys are using terraform in your shop, please share how you manage your workspaces and collaborate within your team.

https://redd.it/eg53rs
@r_devops

What do you use for infrastructure CD

Hey there. I'm just wondering what everyone is using for automatically updating infrastructure. I'm currently trying to get all my provisioning and configuration as code, and while it works great manually, pipelining it all together seems a bit of a task, especially with lots of inputs and outputs of different tasks. We currently have a multiple Jenkins pipelines, with the "main" one that provisions a k8s cluster looking like the followings:
- Terraform to provision VMS. This gives us a tfstate file, which also gets outputted to Jenkins as an artifact.
- Ansible to configure each VM, using Inventory from our tfstate, using the Terraform inventory plugin. This runs multiple playbooks but not all, because we need to...
- Bootstrap Rancher via terraform, which is running as a lone container. This will give us a Rancher URL and token, which can be used to...
- Create a K8S RKE cluster via Terraform. This will give us a cluster URL and a Token which can be used to...
- Install Rancher Agent on each VM via Ansible

As you can see, lots of input is getting piped back and forth, with no easy way of seeing what is and isn't working, without looking at the text output from each job. Does this seem right to you guys? Ideally we would have one big playbook which would do everything, but Ansible and Terraform both have different strengths at different things. All the current CICD tooling seems to be really dev focused

https://redd.it/eg155y
@r_devops

Associate Devops role or stay as a sys admin?

Hi guys,

​

Recently been offered a role to an associate devops for a msp. It is what I have been looking for, except, its a junior role and the pay is slightly lower than mine (8k aud less, I am on 98k)

I am a sr sys admin, but my work dont have much to do with clouds and they lack the cloud vision to moving forward as a company. The offered company has been cloud Consultant in aus for some time now, so they have some big clients like major banks and Airlines.

What would you guys do? Stay as a sys admin / potentially becoming a SE or go to a junior devops role and move on from there?

https://redd.it/eg42m5
@r_devops

elassandra questions

Hey guys,

My company and I are looking into setting up elassandra to be one of our data stores. We want to start small, so what would the difficulty be in setting up a single elassandra cluster? Also, are there any good resources out there besides the docs?

https://redd.it/efz9ek
@r_devops

write ansible playbook that outputs each configuration items in a format that can be imported into a spreadsheet in tab-delimited format

I need to write a playbook an output printing in csv file using python. The output is the ip address config.

https://redd.it/efykws
@r_devops

What’s in store for Kubernetes Operators in 2020?

Here are some thoughts on how Kubernetes Operator movement will evolve in the coming year.

[https://medium.com/@cloudark/whats-in-store-for-kubernetes-operators-in-2020-98263aa23fb9](https://medium.com/@cloudark/whats-in-store-for-kubernetes-operators-in-2020-98263aa23fb9)

Will be happy to discuss any thoughts or questions that you may have.

https://redd.it/egb7as
@r_devops

Using Graylog for doing cool dashboards

Hey buds! I'm kind of newbie in the devops world, so now I'm using Graylog in order to know what it is happening with the infraestructure.

I'm trying to create some histograms with the response codes of my server, but Graylog show me all the messages since 1970! I read that I should add some boundaries but I dont know how to do it. Does reddit can help me? 😁😁

https://redd.it/efy0uw
@r_devops

write ansible playbook that outputs each configuration items in a format that can be imported into a spreadsheet in tab-delimited format

I am trying to write an playbook to parse the output in a csv file using python. Each output is an ip address config. i need the output to be send a csv file. I tried youtube videos but it does not help me.

https://redd.it/efxrfb
@r_devops

Pay to play

A company in Germany asked me to solve a coding problem during a job interview process, but they want me to host a complete API at my own cost at GCP.

Writing a program to demonstrate I know what I am doing is one thing, but it seems they ask a little bit too much. Especially the part where I need to pay money just to apply to a job.

My first thought was "Have they lost their marbles?".

What do you think?

https://redd.it/efv7ns
@r_devops

Migrate large size or Mongodb and MYSQL to another server (low bandwidth)

Hey guys,

Any tips about migrating/cloning large size of mongodb databse and MySQL (100gb++) from one server to another.
Also, I’m on AWS and their instance are on China cloud services, so the network connection could be unstable.

https://redd.it/efwtwl
@r_devops

Hashicorp Vault for temporary aws/gcp/az web console logins?

Hey
I work for a team that often has to give Cloud related demos, workshops, etc.. at schools or other organizations. The demo accounts we use for this are usually created manually now, but we'd like to provision these with an expiration date automatically so we don't have to think about credentials being left active.
These are fairly low level workshops so all we need is an ability to generate a few accounts with Console access and limited permissions to spin up a VM etc. On all 3 major cloud platforms.

Hashicorp Vault seemed to be a great platform-independent tool at first glance but it looks like it is more aimed at provisioning service accounts.
Can this be done Vault? Is there a better tool for our use-case?
Thanks

https://redd.it/efvk7s
@r_devops

Single dev startup: Moving from Heroku to ???

The company I work for runs everything in Heroku. It does a good job and allows us to focus on building stuff, but now that the user base is growing so are the costs... So we are considering other options.

For context, our backend consists of a pretty standard Django/DRF monolith plus Celery, backed by Postgres and Redis. I am the sole backend dev and have very ilttle devops experience, so I want to avoid something overly complex or that will use up too much of my time. I am comfortable with Docker, but not with Kubernetes/swarm/etc.

A self-hosted Heroku would be the holy grail I guess, but I don't think it exists. I've used Dokku before and it works great, but doesn't provide clustering/HA. Flynn is similar and supports clustering, but the docs aren't great and the database support is described as "experimental".

So I've gone on to think about more complex alternatives. Maybe a cluster with docker swarm? Maybe managed Kubernetes from GCP or DO? Maybe Beanstalk or similar?

Has anyone made a similar transition or have any ideas on how to do this without embarking on a massive learning journey/taking up all my time?

https://redd.it/efsjo0
@r_devops