What are the differences between AWS services and AWS marketplace?

Well as stated in the title I want to know what are the differences between an AWS service and AWS marketplace?

For example, I created an AWS Elasticsearch service (which I used Terraform to configure everything including VPC, IAM, etc) and now I found an Elasticsearch from Elastic in the marketplace (https://aws.amazon.com/marketplace/pp/B01N6YCISK?qid=1611156757741&sr=0-1&ref\_=srh\_res\_product\_title) and another one from Bitnami (https://aws.amazon.com/marketplace/pp/B00T6L24IU?qid=1611156682493&sr=0-1&ref\_=srh\_res\_product\_title). I want to know what are the differences? Pros and Cons of each method? and Which one is easier to setup and maintain?

https://redd.it/l1blz8
@r_devops

Generate Yaml files with typescript

Hello everyone,

my team is building extremely complex GitLab CI pipelines and because we are using cdk (typescript) for IaC, I'm thinking if there is some tool/framework which would enable us to write our pipelines in typescript and compile them to Yaml.
Has anyone ever encountered something like that?
Python or golang would also be fine.

Thank you!

https://redd.it/l1eidu
@r_devops

How can it be done

I have around 250 repos in our organization. Due to compliance issues I am the only one who can merge code to master. I collect the pull requests from 10 teams and I merge it. There are a few criterias that I have like I have to merge the PR in a the way it was created and a few dependancy as well like few programs has to be merged before going on with the other merges and I have to wait for the Jenkins to compile the program and then only have to merge the next one . Is there any way we can automate this?

https://redd.it/l0tjtq
@r_devops

CMV: Jenkins shouldn't be anywhere near a robust, mature application environment

Jenkins is hugely flexible, but is very dependent on manual configuration and installation of modules. It's too easy to lose institutional knowledge on how to maintain it well over time especially as teams grow. It seems ideal to small environment, small team setups but that's it as it tends to become yet another pet in what needs to be a cattle oriented paradigm. TeamCity is no better. Am I wrong?

https://redd.it/l1f0se
@r_devops

Advice on starting a devops/cloud business

Hello,

I often see people on here talking about how they run their own dev ops or cloud consultancies and I was wondering if any of you would be willing to help me make the transition from employee to employer.

First, a little about me:

I started in a junior position as a 2nd line technician 10 years ago, moved to a senior technician, then went on to become a network engineer for about 5 years. In 2018 the company I worked for started using public cloud technologies and I started working directly with Azure, AWS and we’ve recently been using Cisco ACI for private cloud as well. I have experience with ARM, Terraform and I’m currently doing a Python automation course. I have been designing, building and maintaining public and private cloud environments for nearly 3 years for thousands of end users.

I know I have marketable skills and I know I can provide value to customers, but I don’t know where to start when it comes to starting my own business. My initial concept to is to run a consultancy that is entirely remote and sells cloud services to help build, maintain and assist customers in moving to remote working environments. I would like to keep costs as low as possible so that I do not have to risk too much capital up front, but I realise that investment is likely to be necessary in order to make this work. What steps do I need to take to get from where I am now to running a consultancy and reaching a global market?

For example:

What do you find is the best way to get customers in the beginning?

What do you find are the best ways to keep on top of market trends and identify services that you can offer competitively?

What is your preferred way to inform and decide on your pricing structure for services provided?

What kind of contracts, services, or customers are good for a starting business of this nature?

Is it best to use subcontractors rather the employees when you are first starting to expand?

I don’t want to just be a lone contractor so how do you expand quickly without overextending yourself? (I understand that I may need to take on more work than I can do alone, but how do I ensure I can meet customer deadlines when I also need to hire staff or contractors to hit those deadlines? Is it just a case of “things will be close”?)

What obstacles should I expect when trying to move the business from the national market into the global market?

I will also be grateful for any other advice.

Thank you.

https://redd.it/l1ik41
@r_devops

Migration to NewSQL: Sentiments of database practitioners

Hello professionals/practitioners, I would like to request if you could have the time to be part of my research study.

The details are provided from the link below.

https://forms.gle/K9p5ruHEb2YypkQG8

Thank you very much for your cooperation.

https://redd.it/l1cctc
@r_devops

Testing how secrets may be logged by Github Actions

I tested the waters using a test repository. Basically, the repository secrets may get logged in some particular scenarios.

https://carrascomj.github.io/public/article/getting-fired-1/

https://redd.it/l1lbl0
@r_devops

Getting into DevOps story

I did not create this blog post nor do I gain anything for you reading IT: https://www.jesuisundev.com/en/when-dev-goes-devops/

Its a really cool story about a Developer and his team going into DevOps. I really liked it. Many in here might feel the same way going from SysAdm into DevOps.

https://redd.it/l1bu0d
@r_devops

The importance of separated environments

In this blog-post, I'll share the difficulties that I faced during deployments to production when using two accounts dev+stg and prd. These difficulties could've been avoided, by separating all environments to different accounts dev, stg, and prd.

https://dev.to/unfor19/the-importance-of-separated-environments-1gja

https://redd.it/l1b4ku
@r_devops

Is CloudBees Dockerhub/registry notification plugin still supported in 2021?

I've recently downloaded this plugin on Jenkins but there seems to be an issue when I try to Monitor latest image change when I attempt to configure my Jenkins Build.

Upon selecting "Monitor Docker HUB for image changes" and selecting specified repositories will trigger this job "docker/imagename"

I received an error message saying that " JSONObject["org-jenkinsci-plugins-registry-notification-opt-impl-TriggerForAllUsedInJob"\] is not a JSONObject. " upon saving the build.


Does anyone know how to submit an issue regarding this?

https://redd.it/l15bp3
@r_devops

Argo-CD help

Hey,


Hopefully there's someone experienced with Argo-CD who can help me out.


I currently want to achieve the following:

1. Terraform provisions EKS/Kubernetes
2. Deploy Argo-CD on Kubernetes
3. Argo-CD configured to point to internal git repositories and start provisioning all of the resources defined there.
4. After which it will sync with and make changes as necessary and prune anything not defined there.


I would like to be able to delete kubernetes/EKS clusters and for Argo-CD to restore it's previously configured options upon cluster re-creation.


Other than persistent volumes is there another way of doing this?
I see that there is some documentation around declarative setup.
What I would like though is the ability to define applications within the UI and for those changes to be saved somewhere centrally like a git repository without having to define them all manually within config files.


Is there any option with Argo-CD to do this natively with a config option or will I have to script exporting the config and re-import post provisioning.


Thanks in advance

https://redd.it/l0sv2m
@r_devops

Understanding / Creating a pipeline?

Hi All I am a software developer and am wanting to learn more about ci cd and building pipelines/ enviornments. I will be starting a job at a smaller company and will be able to learn and do more things aside from purely coding.

​

I know most repos will have a:

* dev branch
* staging / UAT branch
* production branch

I am wanting help to understand the point of staging.

​

For me, dev is where I push new feature code. Before it can be merged in dev it has to build and test

From there what would be a typical pipeline to staging and then prod? I feel as though its the same thing as dev ... build and test so whats the point in having the same thing twice?

​

Any help is appreciated!

​

Thanks

https://redd.it/l0o79v
@r_devops

CI/CD case study for edge infrastructure with a lot of Raspberry Pis

Hi,

In our project, we have a lot of Raspis and configure them all with Ansible. From time to time we are adding new features and fixing bugs in our Ansible scripts and that makes them more complex. We are using GitLab to review our code, but what is missing is a validation of the syntax and that the code will actually work on production’s hardware.

Our Pis are an integral part of our customers. And that’s not the definition for CD(for me :)). Our pis are not easily accessible in a matter of minutes and are distributed across several hundreds of kilometers.

The full case study at https://turingpi.com/case-study-turing-pi-gitlab-ci-ansible/

https://redd.it/l1yy3w
@r_devops

Helm vs Kustomize

What do you prefer for applying Kubernetes resources? Helm, Kustomize, or something else?

From my perspective, #kustomize vs #helm boils down to Kubernetes patching vs templating. Where do you stand? Do you prefer Helm or Kustomize? Why did you choose one over the other?

\>>> https://youtu.be/Twtbg6LFnAg

https://redd.it/l1z39q
@r_devops

Outsourcing your devops is bad - example blogs / articles?

I kicked off a discussion internally in my company regarding cross-functional teams and how you can't just "make a devops team" and throw all your builds to them because you're going to end up in the same situation as when you throw your builds at an Ops team. I'm arguing in favor of cross-functional teams and team ownership of code and deployments from git to Live.

I have read a lot articles over the years along the lines of "we made a dedicated devops team and it didn't work", but now i want to show those blogs as examples to my team and i can't find any. The most i've got is the DevOps Topologies site.

Does anyone have examples of articles elaborating on the above that I can forward to my team for further reading?

Thanks very much.

EDIT: I've posted a summary of my findings below. Thanks all for the help.

https://redd.it/l1vejo
@r_devops

Generic Declarative Infra testing framework

I might just be struggling with what to google here so any pointers are useful.

I'm looking to spin up temporary infra (EC2 VMs), configure them and then run a suite of performance tests on the infra and spit out the results before tearing down the infra.

In some cases I might want to modify the infra post setup, e.g. bring down a node of a distributed stack.

I'm currently using terraform for provisioning, Ansible for config and have the test suite cases identified in simple scripts. I'd like to build this all into one flow e.g. a) run terraform&ansible b) run test suite.

Are there some declarative testing frameworks I should be looking at for this purpose? I'd rather not write out lengthy code or jenkins jobs.

https://redd.it/l259zd
@r_devops

CloudFlare vs Azure vs Cloudfront vs CDN77 vs Fastly

Hi everyone, sorry if this is not allowed or not the right place to post but I was wondering about the differences between these CDN platforms and what the pros and cons of each might be.

Specifically how CloudFlare compares to these other products.

Many thanks!

https://redd.it/l21bt0
@r_devops

Is it time that we solved building multirepo applications once and for all?

After 9 months working from home I'm starting to become more restless. I haven't had the energy up until now to work with my pet projects that I never complete due to way too grandiose ideas that are totally out of proportion. But now I'm ready, now I will start spending a few evenings every week to work on something new, something that is needed by the community, something at a scale I can actually complete. I want to give back something to humanity, it doesn't matter if it's small and not very technical, but I want it to be valuable to people and make the devops world a better place.

So I was thinking of what I'm good at and what I understand well enough to not make something that's irrelevant without me knowing about it until it's finished. I happen to be pretty good at making build pipelines, I'm talking about the real deal with merge requests, integration tests, dependency chains, reproducibility, etc.

The tool I had in mind would be a reboot of the build system I made for my current client which I've been working with/on for a few months. It builds a couple of embedded Linux platforms, a set of applications that can run on said platforms, and finally a couple of complete firmware images put together by slapping a subset of the applications on top of one of the Linux platforms. I want to generalize this to be able to build whatever you like such as node or python applications, not just embedded linux applications.

Some of the features I can think of from the top of my head.

1. Fully reproducible, check out the meta repo from 14 years ago on any computer be it windows, Ubuntu, arch or nixos and hit build. No dependencies required other than installing one single binary and you'll get the same artifacts as you got 14 years ago down to the last bit.

2. Smart caching, only build what have never been built on a build server before, the rest is brought in from cache.

3. Distributable builds, you can add as many machines as you'd like.

4. Coherent, run the exact same command to build on your dev machine as you run on the build servers.

5. Highly integratable, use it with github actions, gitlab ci, jenkins, Travis, drone, circle ci, or just locally on your own machine.

Would you be interested in a canned solution such as this? If not, why? If yes, what would you want to get out of it?

https://redd.it/l280nm
@r_devops

MySQL Configuration Performance Tuning Workflow

Every database system operation on a server will have four main system resources. The CPU is the powerhouse behind the system. The memory encodes, stores, and retrieves information. Disk I/O is the input and output process for data moving from storage to other hardware components. The network consists of the client connections to the server.

When using of these resources aren’t optimized, they can cause performance degradation of the operating system and database system. Fine-tuning the parameters of MySQL is vital for DBAs and DevOps engineers that want to prevent and quickly solve performance issues causing SQL server slowdowns. Ultimately the most important metric, is how quickly the query is received and the data returned by the server. The following results can be achieved for database systems with tuned and configured system parameters:

* Improve application performance.
* Improve the efficiency of server resource utilization.
* Reduce costs.

## Adjust MySQL parameters

Difference situations and events may require recalculating and adjusting MySQL system parameters. Instead of a DBA spending resources troubleshooting server performance issues.

We recommend adjust MySQL parameters in the following cases:

* First-time server setup.
* Low applications performance.
* Changing of server resources (RAM, CPU).
* Changing the application or application load like the count of visitors.

## Tuning MySQL Parameters

To get the most out of your server and curating the ultimate workflow we created a step-by-step guide to doing so: [https://dev-to-uploads.s3.amazonaws.com/i/7994irnb3xtds8199s8b.png](https://res.cloudinary.com/practicaldev/image/fetch/s--I91oCgLs--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://dev-to-uploads.s3.amazonaws.com/i/7994irnb3xtds8199s8b.png)

## 1. MySQL Documentation

MySQL has excellent documentation resources useful to even veteran database system administrators. MySQL provides server reference manuals for each currently supported version.

* [MySQL 8.0 Reference Manual](https://dev.mysql.com/doc/refman/8.0/en/)
* [MySQL 5.7 Reference Manual](https://dev.mysql.com/doc/refman/5.7/en/)
* [MySQL 5.6 Reference Manual](https://dev.mysql.com/doc/refman/5.6/en/)

MySQL 5.6 will not be supported after February 2021.

It’s helpful for all users of MySQL to be familiar with these resources. And it’s highly recommended to spend time working through the documentation to better understand how the database server works.

## 2. Study MySQL Configuration Best Practices

There are an array of resources available to learn about MySQL configuration. MySQL has hundreds of configuration options, but for many server needs, only a handful are critical. The tuning will vary depending on workload and hardware, but DBAs that familiarize themselves with best practices (for their specific version of MySQL) will better be able to understand and solve performance issues.

Releem has collected an [amazing list of articles and resources](https://github.com/Releem/awesome-mysql-performance) that are related to MySQL / MariaDB / Percona configuration.

## 3. Analyze Monitoring Data

Next, monitoring software should be used to continually monitor and analyze data from the MySQL server. These tools will help monitor serve health while providing unique ways to visualize metrics and handle alerts. Open-source as well as licensed software is available. Below are some of the most highly recommended options:

* [Zabbix](https://www.zabbix.com/) is an open-source monitoring tool capable of monitoring networks, servers, cloud, applications, and services. Zabbix is highly secure and easily scalable. 
* [Prometheus](https://prometheus.io/) is open-source monitoring software marketing it’s simplicity and visualization tools. 
* [Percona Monitoring and Management](https://www.percona.com/software/database-tools/percona-monitoring-and-management) is an open-source monitoring solution aimed at helping improve database performance and improving data security
* [Nagios

XI](https://www.nagios.com/) is a premium monitoring software but offers a free trial for new users. Nagios XI promises to be limitlessly scalable and highly customizable. 

Using any of these monitoring tools will provide critical data on how MySQL uses server resources and history values of MySQL status.

## 4. Analyze MySQL Status

Make sure the server has been running for at least 24 hours. Analyze MySQL status to detect any variables that need configuration. Querying with "SHOW GLOBAL STATUS;" will deliver various metrics.

## 5. Use Scripts to get Configuration Recommendations

Using an automated script will save the incredible hassle and time of searching through the metrics. A script will check the metrics from the server and compare them against expected reasonable values. Anything operating outside the expected values will be flagged for review.

* [MySQLTuner](https://raw.githubusercontent.com/major/MySQLTuner-perl/master/mysqltuner.pl) is a powerful configuration tuner that is open-source and available on [Github](https://github.com/major/MySQLTuner-perl).
* [MySQL Tuning-Primer.sh](https://raw.githubusercontent.com/BMDan/tuning-primer.sh/master/tuning-primer.sh) is another strong open-source option on [Github](https://github.com/BMDan/tuning-primer.sh). This tool is older and should be used only with MySQL versions 5.5 - 5.7.

## 6. Calculate Values of MySQL Performance Settings

Now armed with the monitoring data, MySQL status, and configuration recommendations from steps 3 through 5, the specific changes and values can finally be determined to minimize bottlenecks and inefficiencies. The MySQL documentation and configuration best practices that were studied in steps 1 and 2 will show their value during this process. Refer to the documentation, resources, and internet as needed.

## 7. Create New Configuration File

When MySQL is first installed a standard configuration file is installed in the base directory. The new configuration file can be created to reflect the parameter changes made based on the previous steps.

## 8. Apply New Configuration File

The new performance settings can be applied incrementally to test stability and performance or the configuration file can be applied to update all changes at once. The server should now be optimized and ready for queries!

## Streamline the MySQL Tuning

Without experience, fine-tuning the parameters of MySQL requires a significant investment to learn the surrounding information. DBAs and DevOps engineers need to understand the process otherwise, it’s an inefficient use of time. Thankfully MySQL database server tuning doesn’t need to be completed very often. But expect to revisit this process when a new server is setup, performance issues arise, system resources are changed, or an application is changed.

*Originally posted at* [*https://releem.com*](https://releem.com/blog/tpost/op7am2y9b1-mysql-configuration-performance-tuning)

https://redd.it/l23l8n
@r_devops

Request: Guides for ‘RTL’ in Cloud

Looking for any good guides, books, videos on the best way to perform an RTL in a cloud.

Anything to do with migrating from RTLs to cloud would also be appreciated.

https://redd.it/l23amy
@r_devops