what is Lightweight, self hosted CICD in 2021 ?

what is the

lightest
simple
self-hosted
easy installable

CICD ?

https://redd.it/q4pjtw
@r_devops

Self-service automation portal - recommendations?

Looking for recommendations of self-service web portals with approval flows to front automation scripts/tools.

Typical use-case: developer wants to request a new integration environment. They go to a web portal and login with SSO. Choose "new environment" from a catalogue of options and fill out some info. Request is emailed to manager for approval of the cost. Once approved the portal calls our CI which executes terraform and other scripts.

Looking into ServiceNow automation which is functionally perfect but big $$$.

Looked into Ansible Tower but the self-service portal seems quite new and basic.

Anyone got any recommendations on tools that work for you?

https://redd.it/q4u9d2
@r_devops

What agile methodology (if any) do you prescribe to for your DevOps work?

My company runs the development team through a Scrum-based methodology for development (this is basically required for our app anyway). I do most of my work within the development team so I haven’t really questioned my DevOps work also being part of Scrum at least on the ticket side.

But I’ve recently realized it doesn’t truly prescribe to the Scrum methodology as new tooling/update to pipelines/logging are “released” as they’re finished and tested, not on the sprint cycle, so it’s more of a Kanban approach. To me, that seems better suited to the DevOps cycle.

What do you use? And if you’ve used both, what advantages/disadvantages did you see between them?

https://redd.it/q4x2zn
@r_devops

deploying code to bare metal fleet

We have a growing bare metal fleet that we want to automate deployment of our python and go code via our pipeline.

Ideally with a push button deployment, rather than fully automated.

Currently using ansible, but maybe that is not ideal.

I have used ansible AWX in the past, but keeping up with the versions and constant breakages is painful.

https://redd.it/q4vkic
@r_devops

is it possible to use gitlab cicd to trigger systemd commands



I have gitlab runners selfhosted. How to use gitlab CICD to trigger systemd commands ?

Here im running tasks which will run for about 10 hours.

main purpose of using the gitlab is to view logfile in realtime.

https://redd.it/q4pies
@r_devops

Cloud computing research

Hi All, I'm currently in the midst of undertaking research to support my dissertation around cloud computing and would be extremely thankful if you could spare a couple of minutes to complete my survey: https://docs.google.com/forms/d/e/1FAIpQLSen4KxJQKehZfpTZf-rGOqTN7ArftJWxBADBZSlOCyDKSYNAA/viewform?usp=sf\_link

https://redd.it/q4i3f8
@r_devops

Manage multiple Docker hosts/platforms with centrally managed dashboard using portainer

Portainer is a Web User Interface (WUI/GUI) dashboard tool, which can be used to monitor and centrally manage a docker platform. With Portainer, We can add/login to a registry, Manage multiple hosts, Manage hosts on different networks, Build a Container (Docker) Image, pull and push an image from/to container registry, Manage running/deployed containers, deploy a stack/container, Manage accounts, and many more.

Read more about it in link below
https://link.medium.com/8NC5davudkb

https://redd.it/q4mzlo
@r_devops

Redis EC2 vs Elasticache

I know this question has been asked in the past in this sub, but still need some clarification.

Want to optimize my e-Commerce store. DB is optimized with indexes (not sure how else I can optimize it)

Currently, session data is stored in DB which is cumbersome, So was looking to Redis to store session data, images, blocks etc. We have thousands of images.

I have studied Redis, and I understand it purely relies on Ram, but I don't properly understand it enough to know if it requires extra storage space. I am thinking of using redis server on docker/EC2 but not sure if it will eat up on EBS space eg storing cached files?

Also how much ram is enough I see options ranging from 500 MB to 1 GB which seem very little. Our traffic is roughly 2000-3000 users a month (But might scale up later)

Does it make sense to set up redis server in an EC2 instance with 2vCPUs and 4GB RAM? I read that Redis always needs one thread running so is that bound to affect my server performance for the actual serving of files.

https://redd.it/q4gjod
@r_devops

Until 6th of November Google Cloud Standalone courses, Specialization, Professional Certificates are free on Coursera

To see the guidelines, please go through this article

https://redd.it/q556w7
@r_devops

Policy as code for Kubernetes using Shipa and Terraform

https://renjithvr11.medium.com/policy-as-code-for-kubernetes-using-shipa-and-terraform-e6c4f874b824?sk=b7f34fb5bfbab337b280498307e16a8d

https://redd.it/q5ask3
@r_devops

How can I have my application data monitored in grafana to both grahana and kibana through elasticsearch ?

I wrote the below python app that will monitor my internet download and upload speed. (only shared partially)

"""Perform internet speed check.
"""
import os
import re
import logging
import subprocess
import shlex
import getpass
from pprint import pprint

# Internal imports
from home_iot import core as _core

_log = logging.getLogger(__name__)

# create console handler and set level to debug
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)

# create formatter
formatter = logging.Formatter("%(asctime)s;%(levelname)s;%(message)s",
"%Y-%m-%d %H:%M:%S")

# add formatter to ch
ch.setFormatter(formatter)

# add ch to logger
_log.addHandler(ch)


def command_exists(command):
"""Check if command exists or not.

Args:
command (str) : command to run installed in /bin folder.
"""
rc = subprocess.call(shlex.split(command))
if rc == 0:
return True
return False

def _execute_command(command):
if not command_exists(command):
raise ValueError("Command {} not found.".format(command.split()[-1]))
ps = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE)
response = ps.stdout.read()
return response.strip().decode('utf-8')


def check_internet_speed():
"""Performs internet speed test.

Returns:
(dict) : speed test data.
"""
cmd = _execute_command("which speedtest")
_log.info("Command to run: {0} for {1}".format(cmd, getpass.getuser()))
resp = _execute_command("{cmd} --simple --secure".format(cmd=cmd))
ping = re.findall('Ping:\s(.*?)\s', resp, re.MULTILINE)
download = re.findall('Download:\s(.*?)\s', resp, re.MULTILINE)
upload = re.findall('Upload:\s(.*?)\s', resp, re.MULTILINE)
return {
"ping": float(ping[0]),
"download": float(download[0]),
"upload": float(upload[0])
}


def _store_record(data):
"""Stores internet speed test data to db.

Arguments:
data (dict) : internet speed test record.
"""
table_name = "{0}_internet_speed_test".format("lon")
if all(data.values()):
with _core._connect(db="internet_records") as conn:
if not conn:
_log.error("Failed to connect to database engine.")
return
# print("inserting data to {0}".format(table_name))
_core.insert_row(data, table_name, conn)
else:
_log.error("All results not recived.")


def main(dryrun=False):
"""Run the internet speed test and update the record in db.

Keyword Args:
dryrun (bool) : prints to terminal and does not update
record if True else update record and does not print to terminal.
"""
roku_status = None
if dryrun:
data = check_internet_speed()
pprint(data)
return dryrun
else:
data = check_internet_speed()
_store_record(data)

it will find the speed and update the mysql db table. however it is not ideal I belive to store time series data inside of mysql as access over time is getting slower.

​

Now I want to know how I can instead of uploading internet speed data to mysql can update into `elasticsearch`. Would that require influxdb or I can get data added to elastic search without influx?

I want to target elasticsearch in a way so I can pull it both in grafana as well as Kibana. I am thinking kibana will be useful to monitor internet speed analysis. However My knowledge is wee and I am still learning.

https://redd.it/q5dd6u
@r_devops

Tooling advice: Creating a custom project structure/scaffolding template

I’m working on creating a CI/CD pipeline for a low-code integration platform. We create multiple integrations per month. Most of them are REST APIs, but we basically create all kinds of applications. We want to standardize the project structure (folders, specific files and file names) for the applications, so that the same pipeline can build all of them, no matter what type of application they are.

I’ve been thinking about using Python or Bash to do this. Then I started wondering if there are already tools that can do this for me out of the box. Been looking at both Gradle and Yeoman.

Basically, I need to create a project structure looking something like this:
- <project name> (folder)
- setup (folder)
- config (folder)
- tests (folder)
- source (folder)
- .project (XML file created by low-code platform)

Will any of these tools do this for me? Or will it be better creating a custom Python/Bash script that the developers can run when creating new projects. Since there are no pre-built templates/plugins for our low-code platform for neither Gradle or Yeoman, I’ll have to do a bit of custom work anyway.

https://redd.it/q5fq2c
@r_devops

So who's coming to Kubecon in 3 days?

Finally a physical conference and I'm wondering if there's anyone except for vendors. Heard that Blackhat was a bit of a disaster with most attendees staying in their room and joining over zoom.

Anyone here planning to come ?

https://redd.it/q5cti3
@r_devops

Coder to devops?

My team at work is about to take on a large project of building out a full security platform. I'm currently the only developer on the team, and we are about to hire on some new developers to help with this project. Myself and the new hires will be tasked with the dev ops stuff like dev, CI/CD, deployment in the cloud, etc.

My question is, how can I make the most impact on the team? For some context, I really wouldn't even consider myself a developer. I'm 21 and still going to school part time. I interned with the company this summer and then started full time as the summer ended and my mentor left the company so I filled his role. My main responsibilities up to this point have been writing automation scripts/tools (Python) and administrating a few platforms. Very entry level stuff. Before this I interned at a different company writing automation scripts or building automations in a SOAR platform (all Python). The reason I wouldn't consider myself a developer is because all of this work has been done in a super unstructured setting, I write the code and then push it out wherever it needs to go. I've never done things like write unit tests or build a CI/CD pipeline. I've worked in with git, but even then it has been with just one other teammate who was young like me and we didn't have any guidelines, just push it if it works. I'm worried that as we hire "real" developers onto my current team I'm going to be pretty useless and far behind.

Long story short: what can I do to go from a dude that can write Python to a devops engineer that can meaningfully contribute to a real Enterprise scale project?

https://redd.it/q5eckm
@r_devops

Separating Asset Build from Cloud Assembly Artifact in CodePipeline?

I need to package up some code with Gradle to use in my Lambdas. I would like this all to be in one self-contained pipeline, with a pipeline run triggered by a codecommit action to the specified branch. I have this set up in a CDK pipeline. Currently, I run a custom npm script during the cloud synth step that calls gradle and puts the required zip in a directory where the Lambda functions can find it. This does the job. However, it sort of obfuscates what's going on in this stage. Ideally, there would be a separate code asset build stage. I have experimented with events, S3 Buckets, etc. but nothing quite fits the mental model of Build Zip --> Build Cloud Stuff --> Build More Cloud Stuff in Stages. The first two steps become conflated.

What kind of patterns or stages are other people using in their CDK templates? How can I change my pipeline to better fit my mental model?

https://redd.it/q5j4sf
@r_devops

Learning Kubernetes

What's the best way to look at using Kubernetes and gaining experience? (unlikly to use it in my day job at the moment but see it as a requirement for next job hop in 1-2 years).

Should I just dump into the Kubernetes Deep Dive on ACloudGuru (I have an old LA sub) or is there a better way?

I've got "basic" docker knowledge and a bit of Fargate on ECS experience but not running anything complex simple node or java apps without huge traffic / auto scaling.

I'm happy to send $500-600/mo for a few months on AWS or GCP to learn... I'm keen to understand both the infra side and cicd/deployment side? Like is helm still a thing?

https://redd.it/q5ml11
@r_devops

Need guidance on sample coding projects

Been systems engineer with Devops and aws, azure experience.
My scripting experience is limited to powershell and bash.

Been Watching python and Golang course videos on udemy.

I'm more of a learn it by doing it sorta person

Where can I find sample code projects to learn and implement real world like stuff.

https://redd.it/q5o9ix
@r_devops

How do you connect a user's domain to your server and serve static files for that domain?

I have a node server that has HTML files that a user creates.

If a user wants to connect their domain to those static files, how can I set that up?

I want it to work like other website creators like squarespace or wix, how do they do it?

https://redd.it/q5lmp9
@r_devops

A Collection of the Best Jenkins Tutorials for Beginners

I have made a collection of the best Jenkins tutorials for beginners to learn Jenkins. Jenkins is a continuous integration tool.continuous integration Therefore, learning Jenkins will be beneficial to you.

https://redd.it/q5s28o
@r_devops

How would you identify your company’s AWS infrastructure, so you can map it for documentation purposes?

Hi r/DevOps. I will be starting soon as a DevOps Engineer with a focus on Observability/SRE.

One of my intentions is to be able to map out my company’s entire AWS infrastructure, and document it in a diagram (perhaps along the lines of this: https://www.cloudcraft.co/). This will allow me to one: get a grip of what is out there, and two: in times of failure have a decent starting point of looking as to where this may be, in addition to consulting CloudTrail/NewRelic etc.

My question is: any ideas on resources where I can learn this myself? Perhaps AWS have a service that can achieve this using their management console/CLI? Bonus points for automating this to a mapping solution.

I am also open to those of you who think doing this might not be best to achieve what I am aiming for, and your thoughts on alternatives.

Thanks in advance!

https://redd.it/q5wxu5
@r_devops

UK: what do cool kids use to find new jobs ?

Everybody seems to be jumping ship from the current company and im starting to update my CV.

Last time i did was a few years back so im not sure if there is a must-visit place to search.

Im aware of stackoverflow, linkedin, glassdoor, indeed and the usual top google hits.

&#x200B;

Cheers

https://redd.it/q5y4vm
@r_devops