Being afraid of asking a technical question to my coworkers

I have been hired (mid DevOps) along with 2 juniors at the same time. Currently, I am assigned to one project with one of them and one senior dev. We have some K8s to deploy in AWS.

The second junior has been fired after failing one project where his senior was angry at him all the time just because he was asking a lot of technical questions.

Now I am afraid that I will be fired too even if I have some troubles on my way.

Is it a normal state of the company? Previously I was in a company where everyone was learning from everyone and asking a question was a common thing.

https://redd.it/qhiaue
@r_devops

Future proofing... Realistic things I can do, without having programming experience?

So, Systems Engineer here focusing on networks and voice.

The writing is on the wall for my role (not straight away, probably 5 years or so) as it is now, and will be absorbed into a virtual and automated platform....

I have a looonnggg time left in my career, so what can I do to future proof myself? Bearing in mind these skills will only be built in a "non work" environment, as I currently don't deal with virtualisation or automation. I just don't want to be in a position where I've been de-skilled by technology...

I've done a year each in college of C, C++ and Java, but that was over 10 years ago and never professionally.

I can drive Linux for what I need it to do at the moment, can do basic scripting, have a basic understanding of Python.

I am almost a CCNP also.

What can I be doing to cultivate DevOPS skills, that I can present in an interview, that isn't seen as personal time "fluff". I'm almost certain I won't get experience in DevOPS skills for a year or 2 at least in my current role.

The Cisco Devnet exams seem like a logical choice, but still isn't real life examples.

https://redd.it/qhj5c9
@r_devops

Not allowed to have a cross-functional DevOps team

My team are now DevOps and have been for around 2 years. It has been a transition from the "old" model where Software Engineers developed the software and another team handled infrastructure and operations. We've now moved our systems from on-premise to the cloud and the team are supposed to be DevOps Software Engineers doing both development and operations, including creating, deploying, operating the virtualised infrastructure.

We are a team full of Software Engineers with no prior "Operations" experience and we're getting tired of just doing Operations and not enough Development. We had the idea to ask our line manager to put someone in our team with an Operations focus to join the team to make it cross-functional so the "Software Engineers" could get a satisfying diet of development and the Operations expert could get a satisfying diet of Operations.

My line manager has declined the request saying that our company has a policy of not having cross-functional teams. We are not allowed to have cross-functional DevOps teams in our company supposedly. Instead, every engineer must do both Dev & Ops.

What are your thoughts on this situation and what are your thoughts on advantages/disadvantages of having a cross-functional team so that developers can get their fair share of satisfying "coding"? :-)

https://redd.it/qhk2d7
@r_devops

Grafana

Is there a way to visualise latency for different API endpoints through Prometheus in grafana dashboards?

https://redd.it/qhnpj5
@r_devops

Trigger github action after commenting a specific message

I want my github action to trigger only after a specific message/command is commented on the PR. Is this possible ?

https://redd.it/qhuj3k
@r_devops

Share your DevOps horror stories 🎃👻 🎃

Hey folks, as part of Halloween, share your DevOps horror stories that can be of use to others.

https://redd.it/qhqyhl
@r_devops

Have you managed to make a self-service EC2 portal?

What I mean is this: we have a lot of devs constantly testing things. We were a very small infra/DevOps team until lately and so most of them are used to having admin privileges and just kinda using the dev account as a playground.

The result, unsurprisingly, is dozens of instances with no clear owner, and we're unsure if it's even needed. Most surely aren't.

We're in the process of implementing tagging and other such identifiers, but an ambitious goal of ours is to allow people to spin up instances but with more "guardrails", and using Terraform.

I imagine this can be accomplished with a Flask frontend collecting variables, building a tfvar, and then passing it into Terraform or something similar. But of course that sounds a bit difficult and hard to maintain to say the least.

I wanted to ask if any of you have done this successfully, or if you know of some good - ideally free - software that can do it. Or is it just a fool's errands to try to wrangle unpredictable needs into a template like this? Is this taking "self service" too far in the name of cutting down on technical debt? Will we just make ourselves more debt at the end of the day?

https://redd.it/qhxtlc
@r_devops

The process of declining a git push.

I'm trying to understand the process of automatically declining a push that has failed CI tests.

Is the flow goes like the following?

* someone attempts to push changes
* remote branch sees the change but doesn't accept it yet, routes to CI server
* CI server run tests and based on return code decide if to accept the push or not

If not, what happens after the push? What component in git doesn't accept the push yet, and how does it interact with the CI tests?

Thanks ahead!

https://redd.it/qhvh4p
@r_devops

How do I config php-fpm properly?

Ok, so I checked the Apache configs on the server where I can get websites running and the configs on the website where varnish keeps returning 503 and 500 and I found they were the same. The only difference is php-fpm, but I can't think of the reason why that would be the case.

​

[root@webdev01 \~\]# sudo netstat -plnt

Active Internet connections (only servers)

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name

tcp 0 0 127.0.0.2:800.0.0.0:* LISTEN 1679/varnishd

tcp 0 0 172.31.23.5:800.0.0.0:* LISTEN 1644/nginx

tcp 0 0 127.0.0.1:800.0.0.0:* LISTEN 1620/httpd

tcp 0 0 0.0.0.0:220.0.0.0:* LISTEN 1177/sshd

tcp 0 0 127.0.0.1:250.0.0.0:* LISTEN 1439/master

tcp 0 0 172.31.23.5:4430.0.0.0:* LISTEN 1644/nginx

tcp 0 0 127.0.0.1:4430.0.0.0:* LISTEN 1620/httpd

tcp 0 0 127.0.0.1:60820.0.0.0:* LISTEN 1678/varnishd

tcp 0 0 127.0.0.1:112110.0.0.0:* LISTEN 1155/memcached

tcp 0 0 127.0.0.1:63790.0.0.0:* LISTEN 1072/redis-server 1

tcp 0 0 :::22 :::* LISTEN 1177/sshd

tcp 0 0 :::3306 :::* LISTEN 1315/mysqld

[root@webdev01 \~\]#

​

This is where it's working, and we don't see php-fpm.

​

[centos@staging script\]$ sudo /usr/sbin/php-fpm

[28-Oct-2021 15:17:31\] ERROR: An another FPM instance seems to already listen on /var/run/php-fpm/php5-fcgi-staging01.sock

[28-Oct-2021 15:17:31\] ERROR: FPM initialization failed

​

So it's running on a sock? But for some reason I don't see it listening to a port? Are they different?

​

[root@webdev01 \~\]# sudo service php-fpm status

php-fpm (pid 1455) is running...

​

So it's running.

​

On the server where I can't have it running I have:

​

[centos@staging03 script\]$ sudo netstat -plnt

Active Internet connections (only servers)

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name

tcp 0 0 127.0.0.2:800.0.0.0:* LISTEN 2624/varnishd

tcp 0 0 127.0.0.1:800.0.0.0:* LISTEN 2580/httpd

tcp 0 0 172.31.22.60:800.0.0.0:* LISTEN 1582/nginx

tcp 0 0 0.0.0.0:220.0.0.0:* LISTEN 1290/sshd

tcp 0 0 127.0.0.1:250.0.0.0:* LISTEN 1544/master

tcp 0 0 127.0.0.1:4430.0.0.0:* LISTEN 2580/httpd

tcp 0 0 127.0.0.1:60820.0.0.0:* LISTEN 2623/varnishd

tcp 0 0 127.0.0.1:90000.0.0.0:* LISTEN

3397/php-fpm

tcp 0 0 127.0.0.1:112110.0.0.0:* LISTEN 1268/memcached

tcp 0 0 127.0.0.1:63790.0.0.0:* LISTEN 1061/redis-server 1

tcp 0 0 :::22 :::* LISTEN 1290/sshd

tcp 0 0 :::3306 :::* LISTEN 1422/mysqld

​

​

I looked inside etc/php-fpm.d and found this file:

​

[php5-fcgi-elvis\]

listen = /var/run/php-fpm/php5-fcgi-elvis.sock

listen.allowed_clients = 127.0.0.1

user = elvis

;group = elvis

pm = dynamic

pm.max_children = 50

pm.start_servers = 14

pm.min_spare_servers = 14

pm.max_spare_servers = 25

pm.max_requests = 500

catch_workers_output = yes

request_slowlog_timeout = 8

slowlog = /var/log/php-fpm/www-slow.log

php_admin_value[error_log\] = /var/log/php-fpm/www-error.log

php_admin_flag[log_errors\] = on

php_value[session.save_handler\] = files

php_value[session.save_path\] = /var/lib/php/session



listen.owner = apache

listen.group = apache

listen.mode = 0666

​

And it's almost the same as the one on the faulty server:

​

[php5-fcgi-staging03\]

listen = /var/run/php-fpm/php5-fcgi-staging03.sock

listen.allowed_clients = 127.0.0.1

user = staging03

;group = staging03

pm = dynamic

pm.max_children = 13

pm.start_servers = 4

pm.min_spare_servers = 4

pm.max_spare_servers = 7

pm.max_requests = 500

catch_workers_output = yes

request_slowlog_timeout = 8

slowlog = /var/log/php-fpm/www-slow.log

php_admin_value[error_log\] = /var/log/php-fpm/www-error.log

php_admin_flag[log_errors\] = on

php_value[session.save_handler\] = files

php_value[session.save_path\] = /var/lib/php/session



listen.owner = apache

listen.group = apache

listen.mode = 0666

​

However, I found this www.conf file also:

​

[www\]

group = apache

listen = 127.0.0.1:9000

listen.allowed_clients = 127.0.0.1

pm = dynamic

pm.max_children = 50

pm.start_servers = 5

pm.min_spare_servers = 5

pm.max_spare_servers = 35

php_admin_value[error_log\] = /var/log/php-fpm/www-error.log

php_admin_flag[log_errors\] = on

php_value[session.save_handler\] = files

php_value[session.save_path\] = /var/lib/php/session

php_value[soap.wsdl_cache_dir\] = /var/lib/php/wsdlcache

​

So would deleting this www.conf file solve every problem? Because I am thinking there are additional steps. I just don't have the full picture to know what are the things that I can check and what are the things that are wrong.

https://redd.it/qhxtin
@r_devops

DevOps Bootcamp

Hello!

Anyone took that Bootcamp below?

https://www.techworld-with-nana.com/devops-bootcamp

I saw some of her videos on YouTube and she seems very knowledgeable. I was wondering if anyone could recommend her Bootcamp.

Thanks

https://redd.it/qi2w9z
@r_devops

Open-source Wireguard VPN automation with Wiretrustee

Hey folks,

I've been making a few posts about Wiretrustee on Reddit (mostly channels related to self-hosting), but for some reason never did it here :)

We got lots of positive feedback about the project from individuals that are self-hosting the solution and using a free managed version for private use cases (e.g. connecting RPis, building home networks, private Minecraft servers, etc).

I'd love to hear your opinion about the project. Maybe you'd have some cool use cases or maybe point out something that is missing. I'm also curious about the VPN needs of small/medium IT/Engineering teams.

Your feedback will help to further develop the project.

Shortly about Wiretrustee. The details can be found on Github.

Wiretrustee is an open-source VPN platform built on top of WireGuard® making it easy to create secure private networks for your organization or home.

It requires zero configuration effort leaving behind the hassle of opening ports, complex firewall rules, VPN gateways, and so forth.

Wiretrustee automates Wireguard-based networks, offering a management layer with:

Centralized Peer IP management with a UI dashboard.
Automatic Peer discovery and configuration.
UDP hole punching to establish peer-to-peer connections behind NAT, firewall, and without a public static IP.
Connection relay fallback in case a peer-to-peer connection is not possible.
Open Source.
Could be self-hosted.
Works on Linux, Mac, Windows, ARM devices.

Future plans:

Multitenancy.
DNS
Client application SSO with MFA.
Access Controls.
Activity Monitoring.

Let me know what you think. Thank you!

Disclaimer

I'm the author and contributor to the project.

https://redd.it/qi9hej
@r_devops

100 days plan to learn and upskill for job opportunities in DevOps ( CICD Tools, Docker, Kubernetes, Ansible, Cloud, Terraform, Grafana & more! ).

Are you looking for Job in DevOps career?. Did you decided to upskill yourself and start looking for jobs in Devops roles!. I have created a study plan for you. Check this one let me know if it is feasible for you.

Study 2 hrs a day for next 100 days. The main area of focus would be system administration, programming, devops tools and cloud platform. Most of these below topics are covered in video series . You will find this learnings from introductory to advanced knowledge and is better than books and paid lectures

The breakup is as follows

System Administration : Focused on RHCSA/RHCE -- 15 mins per day

Programming: Learn enough for scripting on Python, Go, Ruby . -- 1 hour per day

DevOps Tools: Jenkins/GitLab, Docker,Ansible, Kubernetes, Terraform -- 20 mins per day

Cloud: AWS/Azure/GCP -- 15 mins perr day

Monitoring: Prometheus, Splunk, Grafana -- 10 mins per day

​

If you are capable, it be wise to learn these 5 topics in parallel or you can concentrate one at a time, complete it and then move to the next one.

Suggestions, feedback, criticism all are welcome.

Ask yourself, Are you serious to become DevOps enginner in 2021 ? If yes, then click the Subscribe button now! and spend quality and consistent time for developing your skills.Finally good luck, well no it's not about luck, more about discipline ...

https://redd.it/qid6ap
@r_devops

Move on or stay?

So, recently got into a DevOps role with a company I've been with 2 years(only devops engineer) . I'd say I'm a strong 3rd line/Senior Sysadmin/light dev, I'm in charge of two companies platforms(which is odd I know but we're in that growing phase) , both fully cloud based, 1 with hilarious amounts of microservices/pipelines etc which I build/maintain. As part of an overall strategy we want to implement more automation with our environments etc which is great. So we got some outside consultantency from a DevOps group and they are planned to be coming in to do the work.

So my thing with each company I go to is I like going in at the time where they're growing very quickly and becoming quite large, I end up with way more responsibility and experience than I would have had being at a giant company. This has worked great over the past couple years and I've reached the point where I would cash out again in terms of experience with the big DevOps push strategy. But I'm thinking that with the 3rd party guys coming in. Is there any point in me being there if they're going to do everything anyway?

https://redd.it/qicpqx
@r_devops

Using PowerShell to interact with REST API's

I have a new post regarding using PowerShell to interact with REST API's.

https://seehad.tech/2021/10/29/use-powershell-to-interact-with-rest-apis/

Crafting the API request relies on reviewing the (hopefully) well documented API body structure and requirements for using an access token and how to craft GET or POST methods.

You can also interact with Azure using its API, here is the supporting documentation: https://docs.microsoft.com/en-us/rest/api/azure/

​

Besides Postman, what other visual API collaboration/testing tools are out there worth exploring?

https://redd.it/qif9lk
@r_devops

How do you manage server credentials and logins for 100s for servers/vps.

So our company develops some products and then to host the products we create VPS. Now we have roughly 100-150 such clients atm and so we have 100-150 VPS to manage. (among them 10-15 will require active work). so how to manage these many VPS efficiently. Currently, I use WinSCP, store the credentials and then login if required. Is there any better and efficient way for this?

https://redd.it/qitc3n
@r_devops

Should password file be scalable?

When hosting an encrypted passwords file that the source-code would access to retrieve passwords/keys (either via Hashicorp Vault or a custom made one in Python), should the passwords file be hosted on a single server which would be referenced by the code (while of course, being monitored, audited, and backed up to a different server), or should it be somehow orchestrated across multiple places to avoid a heavy load on the file system?

It's hard for me to imagine that any code would have to read so much from a passwords file that it'd cause a problem on the filesystem.

(I have thought about an idea where it tries to cache the password, and only if the cached password fails, only then read from the encrypted passwords file, but the question still remain)

Is there some best practice I'm missing?

Thanks ahead!

https://redd.it/qj2d57
@r_devops

Resources for learning Kafka

Do you know any good resources for learning Kafka for a DevOps? Learn the basics of configuring Kafka instances and how it works?

https://redd.it/qj2bgh
@r_devops

Elastic Cloud is really good for the price. My Team's Journey...



If you are a relatively small shop and you don't have a ton of traffic volume I recommend looking into Elastic Cloud. I found that from a cost to manage our own elasticsearch instances in terms of resources and the cost savings we got from centralizing logs + apm + infra metrics in one place to be extremely inexpensive based on what you get.

Our breakdown on datadog pricing was about $2k/month all in one logs, metrics, apm for just our AWS environment. Its $1k/month with elastic cloud which includes twice as many hosts with our on-prem environment because its all resource based. We were able to migrate our on-prem elasticsearch and prometheus instances to elastic cloud. Newrelic would have been cheaper if we were really small because they charge per user. In summary we moved all the following to elastic cloud for $1k/month:

1. Two self hosted elastic search instances. AWS & On-Prem
2. 1 Prometheus Instance (replaced with elasticsearch metrics with datastream & elastic agent)
3. DataDog for \~50 hosts with infra monitoring on all, logging on some and APM on most

I have a few compliants... If you don't have elasticsearch experience to start out with your journey is going to be a pain and they don't hold your hand unless you pay a lot of money. Datadog makes it much easier and their support is more responsive even if you are a small shop. DataDog also has a nicer UI in my opinion. Elastic Agent is also new, so you have to use filebeat if you do *anything* non standard with your logs. Also they have very few integrations compared to datadog / newrelic. We have to write our own webhook interface for some stuff such as opsgenie alerts.

https://redd.it/qj7fz9
@r_devops

Guide to secure a server/vps

What are the resources or guides you would suggest for a developer who needs to set up and secure a web server.

I have basically collected this much:

* SSH
* use cert
* disable root login
* change port (contested)
* fail2ban
* Accounts
* principle of least privilege (use specific accounts for only what their needed for)
* Don't run as root
* Firewall
* only have the minimal ports open (http,https,ssh) using ufw or iptables
* SELinux or alternatives (advanced)
* Orchestration concerns(maybe not related to tile)
* do it over a private sub net
* use ssh even then
* Secrets management
* don't store api keys, or certs on disk if possible and load into memory
* user virtualization to isolate host in case webservers are compromised

​

* Misc
* take an inventory of running services and installed software
* keep only what you need
* Logging/perf monitoring
* email,slack for realtime notifications
* backing up your logs in close to real time (in case of compromise for example)
* Always update
* Secure your individual applications (nginx,db,node etc)
* Advanced
* specific distros like alpine or void or build your own
* way smaller attack surface
* musl lib c.
* busybox

​

Cool references i found are:

* Linode/Digital Ocean documentation (basic)
* Arch Linux docs in general but specifically on security/hardening or other distros
* Alot of stuff in github repos in terms of guides but none are authoritative/guaranteed up to date

https://redd.it/qjc1jw
@r_devops

What's the best way to deal with config drift from GUI usage?

Azure's GUI is good. At least good enough that some devs (including me) simply _forget_ IaC exists and use the GUI to make the small modifications necessary for ops. Maybe a scale up of a database here. Maybe changing some permissions there.
The friction of a new PR to the IaC seems to be so high, that people are just not keeping it updated. Fast forward one year and everything's out of whack and we can't replicate any environments.

The simplest solution to implement is a human-process level one, where we simply exhort everyone to update the IaC when they change something. Clearly that hasn't really worked.

The solution that might work better is a drift detector, and maybe auto-applying IaC so devs are forced to PR any changes to the code. But clearly, the devs don't enjoy applying changes to things using code (since they're human too, and everyone likes GUIs) and I'm looking for something better.

I'm thinking that the drift detector should detect changes and make a pull request to the IaCodebase automatically, for modification and acceptance by the owners - since they already made the changes in the GUI. Perhaps they copy-paste configs to some other envs, and merge the PR.
If they reject the PR, the drift is corrected automatically. If not, no further work is necessary by the maintainers - they don't feel like their effort and time updating stuff on the GUI is wasted.


I've looked at older posts like
- [https://www.reddit.com/r/devops/comments/cgcstz/show\_reddit\_configuration\_to\_automatically\_detect/](https://www.reddit.com/r/devops/comments/cgcstz/show_reddit_configuration_to_automatically_detect/): Not Azure, core reco is just not using the GUI. Not great UX IMO, see above.
- https://www.reddit.com/r/devops/comments/60n5qa/how_do_you_manage_configuration_drift/ - this one is too low level for me, but configuration management DB and drift detectors are a good idea.

Overall, UIs like the ones Pulumi or env0.com provide don't seem to be exactly this either. Env0 is close, but seems like they provide their own GUI for specific things instead of re-use current workflows.

Disclaimer - this might be a problem specific to Azure, were the GUI is good enough to use but Azure's IaC support is bad enough to prevent full usage of tools like Az-templates/TF/Pulumi.

https://redd.it/qjgft1
@r_devops