Survey on Multi/Hybrid Cloud CI/CD systems

Hello there. I am a researcher doing a study on CI/CD system in multi/hybrid cloud environments. I am trying to evaluate if there is a use case for offering multi-cloud ready CI/CD systems. Here is the survey link - https://forms.gle/dLNGSJipWxKKBfVVA. I'd highly appreciate if you can help out with answering this survey. Thanks a lot in advance.

Context

This is a research study being conducted to investigate the multi-cloud/ hybrid-cloud use-cases for continuous software delivery (CI/CD) systems.

What is Multi-Cloud

The future of infrastructure platform is multi-cloud, meaning that users will invariably use more than one type of cloud for their infrastructure needs. This could mean they use a private cloud (on-prem deployments) along with one or more public clouds (AWS, Google Cloud, Azure, etc) or in some cases skip private clouds altogether and use one or more public clouds. Customers tend to avoid deploying all their workloads on the same cloud due to concerns of vendor lock-in and to protect against unfair pricing practices from being locked into one cloud platform.

CI/CD on Multi-Cloud

Given that there will be more than one cloud platform that users will use to deploy their software, it is expected that software they develop will need to be deployed/ delivered onto different cloud platforms, each with their own configuration, policies and semantics.

We are trying to understand if there is a use case to offer this CI/CD support for multiple cloud platforms and simplify user workflows that are manual today.

https://redd.it/rbm8jv
@r_devops

How often do you release to production?

I'm weighing up what I should aim for. I know very subjective, but interested to hear peoples experiences with what worked for them.

https://redd.it/rc4179
@r_devops

Automation of Pipeline Creation

Hello,

I'm a software engineer working on creating a CICD architecture for an organization with over one hundred repositories. All of our applications are hosted on AWS, primarily applications deployed through the serverless framework or fargate.

We are currently using Jenkins for CICD, but want to move away from it due to support for it going away company-wide. We want a solution in place that is self-managed.

Given the AWS centric nature of our org, I primarily explored AWS CodeBuild and CodePipeline as the means for an alternative. However, I've not been thoroughly impressed by the means to automate the process of creating pipelines. I've explored using cloudformation to script the process out, so a one time script that builds out the pipeline for our applications. Though, having boilerplate code in every repository to create the pipeline, and having someone run the script with the correct permissions seems awfully inconvenient especially during our transition over. Given that the vast majority of our repositories use the same deployment process, having this duplicated code seems unnatural and not DRY at all.

Am I missing something, or am I to ambitious to think there has to be a better way to automate this process? The boilerplate code required in each repository for jenkins is incredibly small compared to aws codebuild, where all we need to do is specify which type of deployment the application is. With CodeBuild we have to include a buildspec, the cloudformation template for the pipeline, which you could argue gives more customization for your build, however it also opens up a gateway of misuse and misconfiguration that could lead to issues that would be more difficult to debug and assist other teams with.

I'm not going to pretend I'm an expert in devops, but there has to be a more centralized and automated way of creating these pipelines. It is driving me crazy to think that this much boilerplate is needed and can't be automated in a better fashion. Is this really the approach people take using this platform? Manually creating each pipeline?

https://redd.it/rc3jkz
@r_devops

Am i a devops engineer ?

Hey ya'll. My name is Isaac, 24 years old and i'm from the Netherlands.

Currently i'm working for a secondment, and i've been employed at University of Leiden to do some workplace management by removing their old workplace, and placing a new one (new display (sometimes a dockingstation extra) keyboard, mouse)

But that's not what i want to do, i want to become a devops engineer.

Currently i can work with:

C#, Ansible, YAML, Json, Jquery, Ajax, PHP, CSS/HTML, docker (compose), and kubernetes. I can create a server in Linux, host it from on prem to the internet with a traefik reverse proxy/loadbalancing deployed. I could create a new WAN that has to be used, or configure the local WAN. I can code, script and now how to system management of Windows (xp, 7, 10 and now 10). i can create my own powershell scripts and deploy them on prem, and place my code in a local repository or github.

​

I do some work for customers in which i automate alot of workprocesses which would be done by hand. I could maintain contact with the Dev department and Ops department, but i do not know in what degree you should also have work experience as a manager to become a devops engineer (since you're almost managing two or more teams to adopt a different work mentality) I'm a nerd for sure. I wake up with devops and go to sleep with it. There is nothing else that i would want to become in my career (and i think i'm almost there) but my problem also seems to be i do not have any certificates.

TLDR: i have around 4 years coding experience with different languages all which are used to automate or develop things. I have around 4/5 years IT system management work experience and i would like to transition to becoming a devops engineer (or maybe i can already be considered one)

I've read alot, and the only thing i do not seem to have are the following things that in my opinion hinder me to become a full fledged devops engineer:

1) never managed a development or ops team, but i do know how they operate, and i can bridge them

2) do not have any masters degree, or certificates which prove i'm a engineer

3) never worked as a devops engineer (except for some freelance projects i do for customers i get)

​

someone i spoke with which works at google said that i'm already a devops engineer, and i just need to market myself as such. I could put up my resume, maybe that will give people more food to see what i've have done up till now (school and bussiness wise)

https://redd.it/rb0xfi
@r_devops

Just got an online internship at a local firm. Need some advice.

So its a 3 months training program, with assessments in between each month and after 3 months 2 people will be selected for the job. They gave us 5 people a login to acloudguru and directed us to follow the AWS SAA-02 course.

Everday we're supposed to email our daily progress to 3 people in the training team.

I am completely new to this sort of thing.
What i want to know is, is this the usual way of training new employees?

And am i just supposed to put something like "Chapter 7 EBS: Done" in the progress email? How is this credible progress they cant even track individual progress via acloudguru since we all have the same login.

im so confused because its all online and the only interaction i have with the company is through emails sent by HR.

https://redd.it/ravo0g
@r_devops

What do you use CodeCommit or git?

I mean I have just started using AWS, they have CodeCommit, so I was just thinking if it would be good idea to use CodeCommit instead of open source git?

https://redd.it/rcakp1
@r_devops

Anyone have a small devops project idea?

I just got an old Cisco switch! I’m excited to mess around with it so I was wondering if any of you seasoned pros had any ideas on how I can combine the following:

This handy dandy Cisco 2950 switch

Amazon web services

This old windows 10 pro machine (which I use for hyper-V capabilities)

https://redd.it/rcadd1
@r_devops

Digital Ocean challenge for Kubernetes 2021

Hi everyone I am new to k8s and learning. How do I go about starting this challenge? I am planning to do the Deploy an internal container registry.

https://redd.it/rb0a3n
@r_devops

Building dApps

Honestly speaking, I think devOps projects in cryto is gonna skyrocket by 2021

https://redd.it/raziy9
@r_devops

How to diagnose this repeating log output issue?

Hi, our team recently encountered a strange problem with our kubernetes cluster. "kubectl log -f" when the follow option is used, we get continously repeating log output. On the node, using crictl logs -f gets the same behaviour. I don' know where to look for issues. Does anyone have any suggestion? I can't get any valuable support from our cloud work nodes provider unfortunately. They claim it is the service application's problem.

https://redd.it/raraca
@r_devops

Authentication error using API Gateway

Hello,

I have an AWS API that calls a Lambda function, both built by SAM. When I test it via the console, it is works. When I hit it via a browser, I get:

{"message":"Missing Authentication Token"}

I have tried to use the following URL:

https://XXXXxf6cil.execute-api.us-east-1.amazonaws.com/Prod/POST/[email protected]
https://XXXXxf6cil.execute-api.us-east-1.amazonaws.com/Prod/[email protected]
https://XXXXxf6cil.execute-api.us-east-1.amazonaws.com/[email protected]

The URL for my production stage is:

https://XXXXxf6cil.execute-api.us-east-1.amazonaws.com/Prod

When I go to perform a (successful) test of API via the console, the request is what I have below. So basically, I thought that I could add the request on to the end of the production stage URL and use that?

/films?[email protected]

​

I haven't configured any authentication so I suspect the problem is elsewhere? There resource policy on the lambda function looks like it has been generated and applied correctly by SAM. This (films/POST) is currently the only method for this API. If I look at the Method Request, Authorization is set to "None".

https://redd.it/rce80r
@r_devops

AWS vs. Azure for dockerized auto-scaling applications

Hi there,

Our client has proposed that we should look over switching from Azure to AWS, but I want to make sure that this is a good choice. The applications are Python based (Django) Applications that run in Docker, some are just web-servers, some are for handling CRON-tabs. Served via NGINX and uWSGI and CI taken care of by Github actions.

I have been working with Azure for about one month, and I have run into so many quirks and issues that I've almost lost my sense of sanity sitting with their solutions. It's half baked, documentation isn't very good and error logging is catastrophic at times. Short and sweet - I hate it wholeheartedly and want to throw it into the fires of hell every other day, so I'm inclined to go towards AWS - but wanted to check with you who maybe have experience from both/the particular experience of our stack in AWS who can tell how it works for you.

Looking forward reading your replies. Thanks.

https://redd.it/ra7ihh
@r_devops

Don't Just Track Your Machine Learning Experiments, Version Them - Distributed Versioning vs. Centralized Tracking for ML Experiments with DVC and Git

Machine learning experiments often get split between Git for code and experiment tracking tools for meta-information - because Git can't manage or compare all that experiment meta-information, but it is still better for code.

The following guide explains how to apply DVC for ML experiment versioning that combines experiment tracking and version control: **[Don't Just Track Your ML Experiments, Version Them](https://dvc.org/blog/ml-experiment-versioning)** - Instead of managing these separately, keep everything in one place and get the benefits of both, like:

* **Experiments as code:** Track meta-information in the repository and version it like code.
* **Versioned reproducibility:** Save and restore experiment state, and track changes to only execute what's new.
* **Distributed experiments:** Organize locally and choose what to share, reusing your existing repo setup.

Experiment versioning treats experiments as code. It saves all metrics, hyperparameters, and artifact information in text files that can be versioned by Git, which becomes a store for experiment meta-information. The article above shows how with DVC tool, you can push experiments just like Git branches, giving you flexibility to share experiment you choose.

https://redd.it/rcgaio
@r_devops

Best DevOps courses (between beginner and advanced)?

I heard good things about Stephane Maarek's DevOps course, but I'm afraid it's a little too specific for me, since it includes AWS only, and the IaC part is with AWS CloudFormation only. But if I can apply the things I learn from that when I'm creating Terraform scripts for Azure, then that's totally fine with me.
And I also already completed Stephane's AWS Developer course. Oh and one more thing, sometimes I felt like the Developer course was too focused on teaching exactly enough to pass the Certification exam, and it skipped explaining some things more in-depth.


But if anyone can suggest a really good, possibly more generic course on DevOps topic, I'd be very grateful. Also it's okay if it's several seperate courses on topics that are neccessary for a DevOps engineer. I have motivation but am having trouble deciding where to start and how..

https://redd.it/rchk6q
@r_devops

Jenkins pipeline for npm project for code deployment to sandbox

Hi can anyone plz help me out with jenkins pipeline for deployment of 2repos in bitbucket to sandbox environment using npm commands

https://redd.it/rciq8l
@r_devops

Git and GitHub Tutorial – Version Control for Beginners

If you're a beginner developer and want to learn about Git and GitHub then this article is for you.

This tutorial will help you understand what Git and version control are, the basic Git commands you need to know, how you can use its features to boost your work efficiency, and how to extend these features using GitHub.

Check it out: https://www.freecodecamp.org/news/git-and-github-for-beginners/

https://redd.it/rchs9r
@r_devops

Is Kubernetes Betamax?

The other day I was reading through some old Reddit posts about the basic question ‘should I use Kubernetes at all?’ and found this post from Reddit user u/trg0819 where they posit that Kubernetes matters because it is the industry standard.

>Betamax in theory did everything VHS did. But one made its way to be the industry standard, and people still using Betamax after that point were not well regarded.

His point is extremely valid: there’s value to using what everyone else is using, totally outside of its individual merits. In that way, Kubernetes is VHS all the way. Every major cloud platform supports Kubernetes, and 50–80% of containerized applications running in those platforms are using it (this article is old and the current rate is probably higher).

But I still have to ask, is Kubernetes Betamax?

I’ll admit that my analogy requires an understanding of an audio-visual format war from nearly 40 years ago, but bear with me: Sony’s Betamax was a superior video format at the time of its release. Betamax creates a denser image in pure analog, with more vertical lines in each frame. You can watch a comparison here.

So why is there a 0.0% chance that your grandparents’ basement is filled with Betamax tapes? There are a number of theories about price and availability, but one simple answer is that the most complex answer is not the best answer for everyone. The cost and complexity of Betamax players, combined with their relatively short play time, meant they lacked mass-market appeal.

How is this connected to Kubernetes? I don’t think anyone could argue that Kubernetes is a simple tool. Last year I saw people hailing a ‘simple troubleshooting guide’ that felt like… anything but. https://learnk8s.io/troubleshooting-deployments

And as we see a severe talent crunch, with teams unable to hire experienced Kubernetes experts, it seems reasonable to say that Kubernetes may just be too complex for mass-market adoption.

Even though most teams doing distributed workloads are using Kubernetes, it’s still true that most of the web requests that are handled are not handled by a distributed workload, so Kubernetes is the default tool for the job, the job is still not fully wide-spread.

Here’s one of the secrets of ‘failed’ formats like Betamax and DAT tape and others: They lived on for a very long time as tools for professionals.

The reason people aren’t using Kubernetes, why they’re sticking with tools that don’t involve distributed frameworks like serverless, is not because Kubernetes can’t do what they need, it’s because they’re intimidated by a tool that, as the original poster said, feels like overkill.

Okay, so in the real world at this point the analogy breaks down. I’ll mention here that tools and services like Mirantis are popular for a reason: a lot of teams are finding a huge expertise barrier to creating distributed workloads in production, and they need both tools and experts to them across that initial barrier.

https://redd.it/rcl7su
@r_devops

How common it is to fail builds due to security vulnerabilities?

We had a fairly mature DevSecOps practice in the previous company I worked for. We had static & dynamic application security testing, as well as container security and software composition analysis tools.

We broke build and deployment pipelines when high severity vulnerabilities were identified, but the inability to release hot fixes to the code impeded the development velocity.

I decided to develop an aging threshold mechanism that allows developers to exclude specific vulnerabilities in a text file, but the caveat was that the pipeline always checked if the vulnerability is aged over 2 weeks. If it was the case, no more exceptions were allowed to deploy.

On top of it, we had a policy to re-deploy the containers every week, so when a deployment failed, it notified the relevant teams that the deployment failed (we didn't have it in the build process though).

Which portions of these practices are adopted in your companies?

https://redd.it/rcl6cs
@r_devops

Anyone have experience with Sumo Logic for log aggregation and analytics?

We currently use Splunk however our renewal is coming up. With an ongoing cloud migration and for lack of a better term “digital transformation”, a few friends have suggested checking out Sumo Logic as a replacement. We are also evaluating Solar Winds and Data Dog. Any feedback on the platform would be helpful. Thanks!

https://redd.it/rcmpiv
@r_devops

Let's make faster GitLab CI/CD pipelines

In my article, I wrote about a bunch of tips to make your GitLab CI/CD pipelines very fast:

https://blog.nimbleways.com/let-s-make-faster-gitlab-ci-cd-pipelines/

Here is the code for everything in the article

https://gitlab.com/daoudi.mohammed/gitlab-faster-pipelines/-/tree/main

I made patches for all the commits. If you want to try it yourself:

git reset --hard 054bc48b
git apply patches/...

https://redd.it/rcnz5x
@r_devops

GoLang for DevOps in general, Lambda Functions/Automation in particular.

Hello folks, I have been using Python and NodeJS here and there to do different kind of stuff on Lambda functions such as inserting something into a DB or zipping and unzipping a file in S3 etc. I don’t find myself excelling at neither of them. I have been digging into GoLang for sometime now and I just love it! I want to excel at it and use it in my day to day life as a DevOps engineer. I’m also not very sure how to expose and teach myself more GoLang that it would help me in my rather specific purpose.

With my limited time I want to be as effective and time efficient as possible therefore kindly looking for your recommendations on how to proceed.

I’d appreciate if you skip a comparison between Python and Go as I want to follow what I’m excited about as its a great motivation to excel at something. Otherwise, I’m very open to your suggestions.

https://redd.it/rcp4pw
@r_devops