Makefile + Docker = CI Pipeline

I created this project [https://github.com/rosineygp/mkdkr](https://github.com/rosineygp/mkdkr). That helps to create agnostics CI pipelines. It can be shipped in any other big player like Github Actions, Giltab, Circle CI, Travis... and more, also you can run it locally as well.

So take a look ;)

https://redd.it/eh02lc
@r_devops

Help a Developer learn DevOps?

Guys, I am a developer. Worked mostly on .NET and Angular projects. Believe it or not, 6 months ago. I did not even know the meaning of word DevOps. My company had on premise applications that were deployed manually on IIS.
It was only recently that I was exposed to DevOps in my new organization.

I am in a desperate need of a course/courses that cover:

1. CI/CD pipelines for .NET and angular projects
2. Creating Tasks in Pipeline
3. Integration with Github
4. Deployment to Azure
5. Docker
6. Kubernetes (optional)

It is very hard for me to read long documentations. I am a very slow reader. It is very tiring and boring process for me to read long documentations. I lose focus while reading. Therefore it would be great if you could stick to video courses please.
Thank you.

https://redd.it/egvrjs
@r_devops

Monthly 'Getting into DevOps' thread - 2020/01

**What is DevOps?**

* [AWS has a great article](https://aws.amazon.com/devops/what-is-devops/) that outlines DevOps as a work environment where development and operations teams are no longer "siloed", but instead work together across the entire application lifecycle -- from development and test to deployment to operations -- and automate processes that historically have been manual and slow.

**Books to Read**

* [The Phoenix Project](https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/1942788290) - one of the original books to delve into DevOps culture, explained through the story of a fictional company on the brink of failure.
* [The DevOps Handbook](https://www.amazon.com/dp/1942788002) - a practical "sequel" to The Phoenix Project.
* [Google's Site Reliability Engineering](https://landing.google.com/sre/books/) - Google engineers explain how they build, deploy, monitor, and maintain their systems.
* [The Site Reliability Workbook](https://landing.google.com/sre/workbook/toc/) - The practical companion to the Google's Site Reliability Engineering Book
* [The Unicorn Project](https://www.amazon.com/Unicorn-Project-Developers-Disruption-Thriving-ebook/dp/B07QT9QR41) - the "sequel" to The Phoenix Project.
* [DevOps for Dummies](https://www.amazon.com/DevOps-Dummies-Computer-Tech-ebook/dp/B07VXMLK3J/) - don't let the name fool you.

**What Should I Learn?**

* [Emily Wood's essay](https://crate.io/a/infrastructure-as-code-part-one/) - why infrastructure as code is so important into today's world.
* [2019 DevOps Roadmap](https://github.com/kamranahmedse/developer-roadmap#devops-roadmap) - one developer's ideas for which skills are needed in the DevOps world. This roadmap is controversial, as it may be too use-case specific, but serves as a good starting point for what tools are currently in use by companies.
* [This comment by /u/mdaffin](https://www.reddit.com/r/devops/comments/abcyl2/sorry_having_a_midlife_tech_crisis/eczhsu1/) - just remember, DevOps is a mindset to solving problems. It's less about the specific tools you know or the certificates you have, as it is the way you approach problem solving.
* [This comment by /u/jpswade](https://gist.github.com/jpswade/4135841363e72ece8086146bd7bb5d91) - what is DevOps and associated terminology.

Remember: DevOps as a term and as a practice is still in flux, and is more about culture change than it is specific tooling. As such, specific skills and tool-sets are not universal, and recommendations for them should be taken only as suggestions.

**Previous Threads**

https://www.reddit.com/r/devops/comments/e4pt90/monthly_getting_into_devops_thread_201912/

https://www.reddit.com/r/devops/comments/dq6nrc/monthly_getting_into_devops_thread_201911/

https://www.reddit.com/r/devops/comments/dbusbr/monthly_getting_into_devops_thread_201910/

https://www.reddit.com/r/devops/comments/cydrpv/monthly_getting_into_devops_thread_201909/

https://www.reddit.com/r/devops/comments/ckqdpv/monthly_getting_into_devops_thread_201908/

https://www.reddit.com/r/devops/comments/c7ti5p/monthly_getting_into_devops_thread_201907/

https://www.reddit.com/r/devops/comments/bvqyrw/monthly_getting_into_devops_thread_201906/

https://www.reddit.com/r/devops/comments/blu4oh/monthly_getting_into_devops_thread_201905/

https://www.reddit.com/r/devops/comments/b7yj4m/monthly_getting_into_devops_thread_201904/

https://www.reddit.com/r/devops/comments/axcebk/monthly_getting_into_devops_thread/

**Please keep this on topic (as a reference for those new to devops).**

https://redd.it/ei8x06
@r_devops

Choosing the best future-proof platform

Hi, all. I'm the sole developer at a small brick-and-mortar business that has been spending the last couple years modernizing and building its IT infrastructure. We have a custom Laravel-based Point-of-Sale system, lumen-based API Gateway, and e-commerce site that are all running under AWS in a VPC environment. Our VCS is git via Bitbucket. Servers are currently separate EC2 instances, with our MariaDB on RDS. Currently, as I am the sole developer, I have been using Laravel's Homestead for my local development, and Laravel Forge to deploy to our staging environment for testing when we make a commit to our master branch. However, for our production deployments I am still performing the process manually at the end of each sprint.

Our basic layout is this: we have a public-facing web server that runs a Vue/Quasar SPA app. That app communicates with a Lumen-based API Gateway that funnels requests to our Point of Sale/API server (which sits behind a VPN) and back. So, that's three EC2 instances we have to maintain that talk to one another in various ways. I do plan to further segment to microservices in the future (i.e., completely split off the API calls so the Point of Sale is just a frontend to the API). But for now, we have it as-is.

I am planning to move towards a more cohesive CI/CD model in the new year, and with that, Docker containers at the very least. That said, I see so many new tools and platforms for orchestrating deployments and I am unsure what the most time-tested and forward-thinking ones are that won't be dead in a year or lose steam. My current plan is to migrate my local development environment to Docker containers, but use Minikube to run them locally, and then deploy to a Kubernetes Cluster on EKS, thereby bringing us down to one EC2 instance that can handle scaling our app as needed.

I know I am oversimplifying things to a large degree here (and I admit I am not a DevOps expert—I have so much to learn), but first, does this seem like a solid plan? Or should I be segmenting our three servers differently/more? Are there other tools or topologies I should consider? And then there is the question of how to handle the deployment to EKS. I am currently thinking of using Bitbucket Pipelines for that portion.

Basically, in addition to making our environment more team-friendly, CI/CD compliant, and future-proof, I also want to level up my own DevOps skills for career reasons.

Any advice or guidance would be most appreciated. Thanks in advance.

https://redd.it/ei6z7b
@r_devops

As a DevOps Engineer currently in a role, what does your typical day look like? If you were to express it as a percentage, how much coding vs ops do you actually do on a daily basis?

I come from an operational background and was curious how much OOP do you do in comparison to daily operational tasks? What task or group of tasks do you find yourself devoting most of your time?

https://redd.it/ei2b9p
@r_devops

Infrastructure advice for a batch cluster

I've recently been put in charge of a cluster that I had no hand in creating, which is primarily used as a batch system (HTCondor as the backend) and I have the opportunity to make some changes to the underlying infrastructure. I'm quite new to the world of devops, so I've spent some time going over the layout of the infrastructure and it is currently set up like so:

OpenStack is the host, with a terraform config to setup the different types of nodes. Ansible, to set up docker, condor, and Nvidia drivers (among other base tools). Then, docker images are built for each type of node to include the required tools for each (worker node may require python ML libs, login node will need very little, etc). And finally, kubernetes is used to orchestrate the images across the nodes.

At the moment, the system works well and it is easy to roll out changes as I simply need to modify a docker image and provision it with kubernetes.

I'm wondering if there are issues with this type of setup though, particularly with the docker/kubernetes layer. Would it be beneficial to gut that layer, and simply provision each type of node through ansible or terraform straight on the VM? What kind of benefits would that have (if any)?

One of the issues that has come up with this setup is if a job requires a container, we run into a container-in-container situation. I have read about how this is possible but advised against, and instead sibling containers should be used. One issue with this though is from my understanding of this setup, the docker containers are an additional layer of security, so if someone was to run an adjacent container, it would be possible to mount the host root directory.

As stated, I'm new to this world and I wanted to see what the wider DevOps community thinks. I don't plan on making any sweeping changes until I am confident in my knowledge (which I am currently not) so if there are any good resources I should read for this kind of setup, feel free to share them as well, I want to learn all that I can. Thank you!

https://redd.it/ei8li5
@r_devops

I have two vpc stack templates that produce two seperate vpc environments. How can I consolidate them into a single vpc stack? (code below)

So I have a root template that deploys two vpc stacks. Like this:

root.template
\-> vpcstack1.template
\-> vpcstack2.template

In each template vpc resources (vpc, security groups, subnets, ect.) are deployed. They both belong to an app stack on their own vpcstack1 goes to app1 and vpcstack2 belongs to app 2. I'm being required by my organization to deploy both apps into one unified vpc stack. Which makes perfect sense. Why do I need 2 vpcs for 2 apps when I can have 1 vpc hosting 2 apps and all their required network resources.

Below is a snippet of my vpcstack2.template

Vpc:
Type: AWS::EC2::VPC
Properties:
CidrBlock:
Ref: VpcNetBlock
InstanceTenancy: default
EnableDnsSupport: 'true'
EnableDnsHostnames: 'true'
Tags:
- Key: Name
Value:
Fn::Join:
- '-'
- - vpc
- Ref: AWS::StackName
- Key: Platform
Value: nexus-repository-manager
- Key: Version
Value:
Ref: NexusRepositoryManagerVersion


This gives me my second VPC and all the resources in them. How can I get rid of the second vpc and take all the network resources and dump them into vpcstack1.template for a unified stack?

https://redd.it/ei6awx
@r_devops

VMware completes the acquisition of Pivotal. What will happen next?

Business Insider reported saying Pivotal is preparing for layoffs ahead of its $2.7B acquisition by VMware, and this sparked a protest by Pivotal employees internally and then went out on the public. As per the report, Employees started circulating an open letter asking VMware and Pivotal leadership to retain the workplace operations team.

When these acquisitions happen, it is usually so much confusing.
What to expect next?

https://redd.it/ehxtv2
@r_devops

Here's an off the wall question. Anyone in here dealing with or providing care for those with chronic or major health issues, what is your opinion of 'operations' in the medical industry?

After two years and about a million dollars spent on my wife's health, I have to say the entire thing is not inspiring confidence.

https://redd.it/ehws4a
@r_devops

IPAM / DCIM for the cloud

My organization maintains a spreadsheet with a list of all our subnet assignments across several dozen AWS accounts. This list is updated regularly by several people and is not always up to date, and overlaps do happen.
As our deployment continues to scale, the spreadsheet will not. So I'm looking for an IPAM solution, preferably something with auto-discover, that we can use to centralize management of limited networking resources.

I looked at Netbox which seems more geared towards physical DCs, and also doesn't have auto-discover capability (or at least I couldn't figure it out). Before I go and write a script to loop through all our AWS accounts and populate all the VPC and subnet data, does anyone have a better solution?

Thanks

https://redd.it/ehz9mg
@r_devops

Logging solutions that intelligently assigns log entries to a user or session context

Historically, I've used [Timber.io](https://Timber.io) for logging across a number of production accounts, and one of my favorite features is the deep integration with Rails (and other frameworks) that automatically assigns log entries to a user or session context, making troubleshooting functional issues much more efficient.


Their customer support has always been slow on the uptake, but in the last 6-12 months, they've become categorically unresponsive. I'm not sure if the service has been abandoned.


Are there any other out of box logging solutions that have this type of functionality? I'm aware you can cobble together a solution using tags, etc, but having a solution with this type of "magic" integration has been invaluable.


Tapping the wisdom of the community, what other solutions have I missed? Or am I overvaluing the elegance of this solution?

https://redd.it/ehxjfu
@r_devops

Not sure if a company needs what you offer enough to make an application?

Having seen quite some end user company IT environments as well as partner IT companies and there recruitment process let me tell you:

A lot of time they really don’t know what they are looking for. They might even have a dev ops team integrated in their other development environments but depending on who you in the end actually talk to is crucial.

I work for a 3.500 employee company with an IT of around 350 central and a distributed about 70 staff developer team in another country.

The team I applied for had nothing to do with dev ops. Now I’m doing a lot of that though, why?

A lot of companies with a big legacy backbone have stiff 10 year old deployment processes, testing etc. in my case there are fully fledged full stack really beastly kind of nerds. But the ERP team I joined did not have any of these state of the art dev ops processes integrated even though they were in their nearest vercinity.


Not enough time to transfare knowledge.


So the ERP team it self with 5 developer, some internal, freelancer, consultant, some admins and db admins with a product owner in an “agile” environment “:) :)” is kind of stuck in 2006. I work there as a developer originally with half my time being dev ops prioritised nowadays.

Not because they want „dev ops“. Or knew they wanted. But not having to rollout manually on a Sunday seems to be nice. With mails attached to rollouts „Mh nice“.

„Oh we can view all these logs from here now, with dashboards? And the 80k free text passwords from the db disappeared as well Mh nice.“

As said I’m not a dev ops guy I’m a dedicated developer. But I feel like the field has much to offer that’s very in transparent to companies.

After some time of integration let’s say half a year later the value always seemed to be completely out of question. It just became the new way to do it, always.

Every team with digital products as results or input of every field would benefit by having a dev ops dedicated guy in them.

At first it does not seem obvious to them though, try to find ways around that, you probably have a very special off skillset to offer. There are a lot of different development teams in most companies. The ERP teams I usually develop for have a very strong fundamental knowledge of how to do ERP.

Like scary knowledge, knowing they manually file transferred there releases for 15 years every two Sundays.

These specialized developers often don’t really care, for good reasons, how to containerize something or orchestrate it. Automatic rollouts, development plans, general utility or other more ops heavy topics a lot of times are completely of the charts for quite some developers.

Like reaaaally of the charts.

I work as a quite jumpy (1-2year interval) ERP developer (NAV) which is modifiable through its own C/AL language.

But my approach to „nice software“ always leads into a nieche position of doing 50% .Net, 40% dev ops and 10% ERP development it self.

Not because I know things nobody knows. But because the team I sit in naturally does know all the things I don’t know so I can know all the things they don’t know for teams up to 15 developers that worked well since 8 years for me.

It always felt like going the way of least resistance in my job but turned out to be the best random nieche to have found.

There is a 25 man .net developer team with a Kubernetes orchestrated enterprise storage management system (feeding our ERP) they have Elasticsearch, Kafka for messaging, fledged rollouts on commis etc.

To the front of the room I sit in. Even though working there as a full time .net dev there would seem nice I can’t imagine it to be, there all the “good shit” already is with me providing much more value in other teams.

With a 100 man java heavy web shop team. Fully automated, everything, state of the art, way to expensive feeding our ERP through questionable interfaces. Duo to web shop developers naturally not knowing how an ERP works internally and an ERP developer naturally not knowing what kind of in

terface clock work is the best for web shop infrastructures the result often are scary scary sql synchs duo to that being the only common ground these different teams can find.

At the end of the hallway 20meter away.

With the ERP team ONLY being able to work *inside* the ERP duo to that being what an ERP developer is. In medium companies with team sizes around 25 developers with 4-5 teams having to work together, the ERP team in 4 predecessor end user companies didn’t think about hiring more .net/other exotic resources until I randomly brought them in my bag with me cloaked as an ERP dev 😎

I feel like there probably are similier naturally good fitting fields of work for specific skill sets. But this combination feels like cheating.

https://redd.it/eiek67
@r_devops

What are some of awesome online tools/apps/services that are free, but require a credit card?

What are some of awesome online tools/apps/services that are free, but require a credit card?

https://redd.it/ehw234
@r_devops

What I learned about Product Delivery in 2019

I've been a long time reader of r/DevOps and noticed that most of the content posted here is on the technical side of DevOps (AWS, k8s, IaC) but very little is posted about the delivery and methodology aspects.

At the start of the year I (a Developer) joined a product team at a large tech company. I knew very little about DevOps and Product Delivery. Here's a write up of the books I've read and things I've learned in this area. Most of these are focused on the philosophy of DevOps and how it relates to developing, shipping, and monitoring products.

> What is Product Delivery? I’d define it as a set of practices, attitudes, and tools which can be used to improve the quality of software and the speed at which it’s delivered to customers. It’s a large field which takes in areas of Lean Product Development, Agile, Software Development, DevOps, and SRE. My approach was to read a lot of books and blogs on these topics and discuss the ideas with colleagues around the office. I’ve written this from the point of view as a developer on a product team at a large tech company. A lot of these ideas are applicable to solo entrepreneurs and those working on side projects.

https://isthisit.nz/posts/2019/product-delivery-learnings-2019/

https://redd.it/eioalg
@r_devops

Using Packer to fetch Ubuntu ISO and convert to template - vSphere -- waiting for IP address issue

As the title states, attempting to use packer, to fetch a ISO, pull it down, configure it with basic items and then convert to a template.

It works up until the very end, when at the packer CLI, it says "Waiting for SSH to become available".When i log into vsphere, I can see the image boot up, run through the installer then gets to the login prompt. Even though it has an IP address, its at the login prompt, packer is 'stuck' waiting or SSH to become available. (i can SSH to the box to verify openssh-server is accepting connections, which it is).

==> Some builds didn't complete successfully and had errors:
--> vsphere-iso: ssh: handshake failed: ssh: unable to authenticate, attempted methods [password none], no supported methods remain

Ultimately what happens is a timeout, then every backs out, the VM is deleted.

I feel I am so close and missing something very very easy and simple.

I am using packer 1.35 and jetbrains vpshere-iso plugin.

Here are my files:

{
"builders": [
{
"type": "vsphere-iso",

"vcenter_server": "{{user `vcenter_server`}}",
"username": "{{user `username`}}",
"password": "{{user `password`}}",
"insecure_connection": "true",

"vm_name": "temp_ubuntu18",
"datastore": "{{user `datastore`}}",
"folder": "{{user `folder`}}",
"convert_to_template": "true",
"cluster": "{{user `cluster`}}",
"network": "{{user `network`}}",
"boot_order": "disk,cdrom",

"guest_os_type": "ubuntu64Guest",

"ssh_username": "{{user `ssh_username`}}",
"ssh_password": "{{user `ssh_password`}}",

"CPUs": 1,
"RAM": 1024,
"RAM_reserve_all": true,

"disk_controller_type": "pvscsi",
"disk_size": 16768,
"disk_thin_provisioned": true,

"network_card": "vmxnet3",

"iso_urls": "https://cdimage.ubuntu.com/releases/18.04/release/ubuntu-18.04.3-server-amd64.iso",
"iso_checksum": "7d8e0055d663bffa27c1718685085626cb59346e7626ba3d3f476322271f573e",
"iso_checksum_type": "sha256",

"floppy_files": [
"./preseed.cfg"
],
"boot_command": [
"<enter><wait><f6><wait><esc><wait>",
"<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
"<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
"<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
"<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
"<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
"<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
"<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
"<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
"<bs><bs><bs>",
"/install/vmlinuz",
" initrd=/install/initrd.gz",
" priority=critical",
" locale=en_US",
" file=/media/preseed.cfg",
"<enter>"
]
}
],

"provisioners": [
{
"type": "shell",
"inline": ["echo 'template build complete'"]
}
]
}

Here is my variables file:

{
"vcenter_server":"xxx.xxx.xxx.xxx",
"username":"XXXXXX",
"password":"XXXXX",
"datastore":iSCSI-storage",
"folder": "_Templates",
"cluster": "crazytown",
"network": "vDEV",
"ssh_username": "root",
"ssh_password": "XXXXX"
}

Finally, my preseed file:

d-i passwd/user-fullname string dude
d-i passwd/username string dude
d-i passwd/user-password password mypassword
d-i passwd/user-password-again password mypassword
d-i user-setup/allow-password-weak boolean true

d-i

partman-auto/disk string /dev/sda
d-i partman-auto/method string regular
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true

d-i passwd/root-login boolean true
d-i passwd/root-password password mypassword
d-i passwd/root-password-again password mypassword


d-i pkgsel/include string open-vm-tools openssh-server

d-i grub-installer/only_debian boolean true

d-i preseed/late_command string \
echo 'dude ALL=(ALL) NOPASSWD: ALL' > /target/etc/sudoers.d/dude ; \
in-target chmod 440 /etc/sudoers.d/dude;

d-i finish-install/reboot_in_progress note

Driving me bonkers because i feel it is simple.

Something simple I am missing? something out of order?

Appreciate the help and extra eyes.

Cheers

&#x200B;

TCG

https://redd.it/eip3ne
@r_devops

Looking for a dashboard for github actions

Hi, I'm used to dashboards surfacing CI/CD status on GitLab, is there anything like that for github actions?

https://redd.it/eioj0b
@r_devops

What are the chances of getting remote DevOp Jobs?

Hey! I am from a small island in the Caribbean, what are the chances of getting an entry level remote job as a DevOp with foreign companies?

My only option is remote work due to my location and opportunities at hand.

Thanks!

https://redd.it/eikjuh
@r_devops

Considering if to continue in the devops field

I guess this is more a personal issue, but an insight might be helpful.

I'm a woman in my mid 30's. Currently, was fired from my job after 3 months. I worked part-time (60%) The mutual understanding is this wasn't necessarily based on technical issues but more fitting issues. I have good skills but I also think they should be improved.
The company is around 20 people, there are 2 devs in Ukraine while the managers are based in my country. I was the only woman in the office.
I felt they don't really had understanding what Devops means and everything was abstract, which made it dificult to accomplish anything.

The reason I work part-time is because Im also doing art and painting. This was a hobby that got extended and it's hard for me to let go.
In my country, it's dificult to find good devops so even when sent resume to full time jobs I still got called and they agreed to consider 60%.
I'll admit that I have mild mental issues like anxiety and depression that I'm handling but it's still exist.
My fear is, that even if i'll still be able to find something else the characteristics made me failed here will be similar in other places. In the long run, I prefer to do a change now than in my 40's.


TL;DR- a woman in mid 30's, artist, some mental issues, still a devops for now. What do you think my options and if I have any?

https://redd.it/eilqta
@r_devops

Need suggestions for contract templates for DevOps side work

Finally getting back into contracting with a job for a former manager to help him set up his infrastructure from scratch for his new business. Last time I did some contract jobs, I didn't really do things right and left myself open to some serious liability. This time around I've established an LLC with a business partner who is handling security related jobs, a separate business account, insurance, and soon a CPA and expense tracking software.

I was wondering if there are any good contract templates that are geared towards this type of work that folks here can recommend?

Any other recommendations on doing this the right way? Thanks!

https://redd.it/eijfsp
@r_devops