trying to implement orderbook/exchange API

Hi everyone, being developer and not devops, i'm trying to implement a system of orderbook with performances, I've come with an idea so far but would like to know what you guys think about it, what tools should i use or any other advices.

1. orderbook as micro-service, with grpc to expose functions

2. my APIs (websocket, rest, grpc), connecting to those orderbook micro-services

3. a Envoy Proxy for the front end to use the grpc API

What do you think about it?

what tool would you use for this in production?
The first idea I have in mind is kubernetes but I might be missing a point.

Thanks for any advices.

https://redd.it/q4h45h
@r_devops

State of uptime monitoring

Hi! I'm doing some research and would highly appreciate your professional opinions. I've been in software and devops for 15 years - from small startups and consultancies to mid-sized SaaS/platform companies. One thing that has always intrigued me was no-downtime approach to infrastructure, be it deployment or chasing uptime nines. No matter how good the team has been, a reliable monitoring solution is crucial when things go wrong. I've used various tools such as CopperEgg, Uptimerobot, StatusCake, Pingdom, New Relic, Datadog etc. but especially when it comes to startups or small companies, I've never had stellar experience with the services for one reason or the other. Especially for the cases where there's no financial backing or a steady revenue stream, and every cent counts.

I'm curious to know, how's the broader experience using uptime monitoring services. Is all well? Is it something you prefer to handle in-house or outsource to a service? Does the standard pricing model make sense? Any thoughts and opinions for this niche are helpful.

Disclaimer: I'm doing market research for various services ( including uptime monitoring) with the purpose of starting a solo venture myself in a niche where there's a clear need for a solution that doesn't exist or improves greatly on what's already out there.

View Poll

https://redd.it/q4gfr8
@r_devops

What to charge for setting up AWS Account + Application

Hi everyone,

first of all: I'm not sure whether I'm posting in the correct sub. If not please notify me and maybe point me to the correct sub if you know.

Now my question: I've been texting with another user on Reddit who represents a small company in South-America. They asked about how to set up a docker-based application on AWS and had some questions regarding AWS ECS. I answered the questions but they pointed out that it is too complex of a setup for them.
Instead, they asked me what I would charge for setting up the account properly, building the infrastructure and help them to get started. They also offered payment and I'm inclined to help them out because it's a new kind of application for me and would be a great opportunity to get exposure and learn.

My question now is: How much should I charge for setting up: AWS landing zone + small application + knowledge transfer? My background: Certified AWS Architect with a couple of years experience and DevOps background.

Also, I'm not familiar with the legal obligations that come with such stuff. Has anyone here some advice for me?

​

Best and thank you for your time

https://redd.it/q4kqf0
@r_devops

Which kunernetes cert to get for an EKS devops engineer?

My company is new to kubernetes, and we’re going to use EKS and my team is going to manage the cluster. I was looking for a cert to level up my game and I thought that the Certified Kubernetes Administrator would fit, but now that I’ve started studying for it, it looks like it focuses on a lot of things that EKS abstracts away like managing etcd. Would I be better off going for the Certified Kubernetes Application Deveoper? Keep in mind my team doesn’t write any code but we are going to help train the devs on kubernetes. I already know my way around the basics of kubectl and helm.

https://redd.it/q4pnvt
@r_devops

Need some insights

Hey, how are you guys?

I'm 24 and currently I work at a company and I'm kinda of a jack of all trades. I am the sole responsible for CI(Jenkins), release, deploy, database structures and I'm also having to code some tools(not the tools that we use to automate our process, but actual portfolio tools that goes to the clients) since the guy who was with me took another offer. The entire company relies on Windows. The devs only do the dev stuff, so I have to guarantee that what they do is delivered to production. Most of the time I hear "not my problem, it works on my machine!!!". If something happens, it is implied that the problem came from me, the Infraestructure guy.

I'm kinda stressed with all this. I started in 03/2020 as a intern, got hired in 08/20, and became this main guy for everything since the other fella left in March this year. Don't get me wrong, I loved and took this opportunity with the most enthusiasm you could even imagine. My bosses probably think that if I can take care of everything, than there is no need to hire another guy to help. They gave me two raises and all that talk of me deserving it, but actually it's because I'm doing two people's jobs by myself. I know that. And the pay is not even that good. And the overtimes, Jesus Christ, sprint every week, people calling me late night because the bosses need something delivered by the end of the day. Its things on top of things that need attention at the same time. I can't say no. Give multiple stuff to me and I will deliver in the next hour. Probably that was my mistake, showing what I can actually do.

I'm trying to learn new things. Got in love with the DevOps culture. But I need to learn so much. I'm a fast learner. I can adapt really well. I started with Linux and Docker, and I know the basics of it. Can I start applying for other jobs that requires it? What are my chances? Are companies willing to get a guy who is ready to learn, with passion and hungry, or to they need those skillsets asap?

I feel like I'm stuck. Knowing a bit of everything won't get me far.

https://redd.it/q4qdix
@r_devops

what is Lightweight, self hosted CICD in 2021 ?

what is the

lightest
simple
self-hosted
easy installable

CICD ?

https://redd.it/q4pjtw
@r_devops

Self-service automation portal - recommendations?

Looking for recommendations of self-service web portals with approval flows to front automation scripts/tools.

Typical use-case: developer wants to request a new integration environment. They go to a web portal and login with SSO. Choose "new environment" from a catalogue of options and fill out some info. Request is emailed to manager for approval of the cost. Once approved the portal calls our CI which executes terraform and other scripts.

Looking into ServiceNow automation which is functionally perfect but big $$$.

Looked into Ansible Tower but the self-service portal seems quite new and basic.

Anyone got any recommendations on tools that work for you?

https://redd.it/q4u9d2
@r_devops

What agile methodology (if any) do you prescribe to for your DevOps work?

My company runs the development team through a Scrum-based methodology for development (this is basically required for our app anyway). I do most of my work within the development team so I haven’t really questioned my DevOps work also being part of Scrum at least on the ticket side.

But I’ve recently realized it doesn’t truly prescribe to the Scrum methodology as new tooling/update to pipelines/logging are “released” as they’re finished and tested, not on the sprint cycle, so it’s more of a Kanban approach. To me, that seems better suited to the DevOps cycle.

What do you use? And if you’ve used both, what advantages/disadvantages did you see between them?

https://redd.it/q4x2zn
@r_devops

deploying code to bare metal fleet

We have a growing bare metal fleet that we want to automate deployment of our python and go code via our pipeline.

Ideally with a push button deployment, rather than fully automated.

Currently using ansible, but maybe that is not ideal.

I have used ansible AWX in the past, but keeping up with the versions and constant breakages is painful.

https://redd.it/q4vkic
@r_devops

is it possible to use gitlab cicd to trigger systemd commands



I have gitlab runners selfhosted. How to use gitlab CICD to trigger systemd commands ?

Here im running tasks which will run for about 10 hours.

main purpose of using the gitlab is to view logfile in realtime.

https://redd.it/q4pies
@r_devops

Cloud computing research

Hi All, I'm currently in the midst of undertaking research to support my dissertation around cloud computing and would be extremely thankful if you could spare a couple of minutes to complete my survey: https://docs.google.com/forms/d/e/1FAIpQLSen4KxJQKehZfpTZf-rGOqTN7ArftJWxBADBZSlOCyDKSYNAA/viewform?usp=sf\_link

https://redd.it/q4i3f8
@r_devops

Manage multiple Docker hosts/platforms with centrally managed dashboard using portainer

Portainer is a Web User Interface (WUI/GUI) dashboard tool, which can be used to monitor and centrally manage a docker platform. With Portainer, We can add/login to a registry, Manage multiple hosts, Manage hosts on different networks, Build a Container (Docker) Image, pull and push an image from/to container registry, Manage running/deployed containers, deploy a stack/container, Manage accounts, and many more.

Read more about it in link below
https://link.medium.com/8NC5davudkb

https://redd.it/q4mzlo
@r_devops

Redis EC2 vs Elasticache

I know this question has been asked in the past in this sub, but still need some clarification.

Want to optimize my e-Commerce store. DB is optimized with indexes (not sure how else I can optimize it)

Currently, session data is stored in DB which is cumbersome, So was looking to Redis to store session data, images, blocks etc. We have thousands of images.

I have studied Redis, and I understand it purely relies on Ram, but I don't properly understand it enough to know if it requires extra storage space. I am thinking of using redis server on docker/EC2 but not sure if it will eat up on EBS space eg storing cached files?

Also how much ram is enough I see options ranging from 500 MB to 1 GB which seem very little. Our traffic is roughly 2000-3000 users a month (But might scale up later)

Does it make sense to set up redis server in an EC2 instance with 2vCPUs and 4GB RAM? I read that Redis always needs one thread running so is that bound to affect my server performance for the actual serving of files.

https://redd.it/q4gjod
@r_devops

Until 6th of November Google Cloud Standalone courses, Specialization, Professional Certificates are free on Coursera

To see the guidelines, please go through this article

https://redd.it/q556w7
@r_devops

Policy as code for Kubernetes using Shipa and Terraform

https://renjithvr11.medium.com/policy-as-code-for-kubernetes-using-shipa-and-terraform-e6c4f874b824?sk=b7f34fb5bfbab337b280498307e16a8d

https://redd.it/q5ask3
@r_devops

How can I have my application data monitored in grafana to both grahana and kibana through elasticsearch ?

I wrote the below python app that will monitor my internet download and upload speed. (only shared partially)

"""Perform internet speed check.
"""
import os
import re
import logging
import subprocess
import shlex
import getpass
from pprint import pprint

# Internal imports
from home_iot import core as _core

_log = logging.getLogger(__name__)

# create console handler and set level to debug
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)

# create formatter
formatter = logging.Formatter("%(asctime)s;%(levelname)s;%(message)s",
"%Y-%m-%d %H:%M:%S")

# add formatter to ch
ch.setFormatter(formatter)

# add ch to logger
_log.addHandler(ch)


def command_exists(command):
"""Check if command exists or not.

Args:
command (str) : command to run installed in /bin folder.
"""
rc = subprocess.call(shlex.split(command))
if rc == 0:
return True
return False

def _execute_command(command):
if not command_exists(command):
raise ValueError("Command {} not found.".format(command.split()[-1]))
ps = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE)
response = ps.stdout.read()
return response.strip().decode('utf-8')


def check_internet_speed():
"""Performs internet speed test.

Returns:
(dict) : speed test data.
"""
cmd = _execute_command("which speedtest")
_log.info("Command to run: {0} for {1}".format(cmd, getpass.getuser()))
resp = _execute_command("{cmd} --simple --secure".format(cmd=cmd))
ping = re.findall('Ping:\s(.*?)\s', resp, re.MULTILINE)
download = re.findall('Download:\s(.*?)\s', resp, re.MULTILINE)
upload = re.findall('Upload:\s(.*?)\s', resp, re.MULTILINE)
return {
"ping": float(ping[0]),
"download": float(download[0]),
"upload": float(upload[0])
}


def _store_record(data):
"""Stores internet speed test data to db.

Arguments:
data (dict) : internet speed test record.
"""
table_name = "{0}_internet_speed_test".format("lon")
if all(data.values()):
with _core._connect(db="internet_records") as conn:
if not conn:
_log.error("Failed to connect to database engine.")
return
# print("inserting data to {0}".format(table_name))
_core.insert_row(data, table_name, conn)
else:
_log.error("All results not recived.")


def main(dryrun=False):
"""Run the internet speed test and update the record in db.

Keyword Args:
dryrun (bool) : prints to terminal and does not update
record if True else update record and does not print to terminal.
"""
roku_status = None
if dryrun:
data = check_internet_speed()
pprint(data)
return dryrun
else:
data = check_internet_speed()
_store_record(data)

it will find the speed and update the mysql db table. however it is not ideal I belive to store time series data inside of mysql as access over time is getting slower.

​

Now I want to know how I can instead of uploading internet speed data to mysql can update into `elasticsearch`. Would that require influxdb or I can get data added to elastic search without influx?

I want to target elasticsearch in a way so I can pull it both in grafana as well as Kibana. I am thinking kibana will be useful to monitor internet speed analysis. However My knowledge is wee and I am still learning.

https://redd.it/q5dd6u
@r_devops

Tooling advice: Creating a custom project structure/scaffolding template

I’m working on creating a CI/CD pipeline for a low-code integration platform. We create multiple integrations per month. Most of them are REST APIs, but we basically create all kinds of applications. We want to standardize the project structure (folders, specific files and file names) for the applications, so that the same pipeline can build all of them, no matter what type of application they are.

I’ve been thinking about using Python or Bash to do this. Then I started wondering if there are already tools that can do this for me out of the box. Been looking at both Gradle and Yeoman.

Basically, I need to create a project structure looking something like this:
- <project name> (folder)
- setup (folder)
- config (folder)
- tests (folder)
- source (folder)
- .project (XML file created by low-code platform)

Will any of these tools do this for me? Or will it be better creating a custom Python/Bash script that the developers can run when creating new projects. Since there are no pre-built templates/plugins for our low-code platform for neither Gradle or Yeoman, I’ll have to do a bit of custom work anyway.

https://redd.it/q5fq2c
@r_devops

So who's coming to Kubecon in 3 days?

Finally a physical conference and I'm wondering if there's anyone except for vendors. Heard that Blackhat was a bit of a disaster with most attendees staying in their room and joining over zoom.

Anyone here planning to come ?

https://redd.it/q5cti3
@r_devops

Coder to devops?

My team at work is about to take on a large project of building out a full security platform. I'm currently the only developer on the team, and we are about to hire on some new developers to help with this project. Myself and the new hires will be tasked with the dev ops stuff like dev, CI/CD, deployment in the cloud, etc.

My question is, how can I make the most impact on the team? For some context, I really wouldn't even consider myself a developer. I'm 21 and still going to school part time. I interned with the company this summer and then started full time as the summer ended and my mentor left the company so I filled his role. My main responsibilities up to this point have been writing automation scripts/tools (Python) and administrating a few platforms. Very entry level stuff. Before this I interned at a different company writing automation scripts or building automations in a SOAR platform (all Python). The reason I wouldn't consider myself a developer is because all of this work has been done in a super unstructured setting, I write the code and then push it out wherever it needs to go. I've never done things like write unit tests or build a CI/CD pipeline. I've worked in with git, but even then it has been with just one other teammate who was young like me and we didn't have any guidelines, just push it if it works. I'm worried that as we hire "real" developers onto my current team I'm going to be pretty useless and far behind.

Long story short: what can I do to go from a dude that can write Python to a devops engineer that can meaningfully contribute to a real Enterprise scale project?

https://redd.it/q5eckm
@r_devops

Separating Asset Build from Cloud Assembly Artifact in CodePipeline?

I need to package up some code with Gradle to use in my Lambdas. I would like this all to be in one self-contained pipeline, with a pipeline run triggered by a codecommit action to the specified branch. I have this set up in a CDK pipeline. Currently, I run a custom npm script during the cloud synth step that calls gradle and puts the required zip in a directory where the Lambda functions can find it. This does the job. However, it sort of obfuscates what's going on in this stage. Ideally, there would be a separate code asset build stage. I have experimented with events, S3 Buckets, etc. but nothing quite fits the mental model of Build Zip --> Build Cloud Stuff --> Build More Cloud Stuff in Stages. The first two steps become conflated.

What kind of patterns or stages are other people using in their CDK templates? How can I change my pipeline to better fit my mental model?

https://redd.it/q5j4sf
@r_devops

Learning Kubernetes

What's the best way to look at using Kubernetes and gaining experience? (unlikly to use it in my day job at the moment but see it as a requirement for next job hop in 1-2 years).

Should I just dump into the Kubernetes Deep Dive on ACloudGuru (I have an old LA sub) or is there a better way?

I've got "basic" docker knowledge and a bit of Fargate on ECS experience but not running anything complex simple node or java apps without huge traffic / auto scaling.

I'm happy to send $500-600/mo for a few months on AWS or GCP to learn... I'm keen to understand both the infra side and cicd/deployment side? Like is helm still a thing?

https://redd.it/q5ml11
@r_devops