Amazon introduces IPv6-only subnets and EC2 instances

Amazon announced about IPv6-only subnets and EC2 instances

https://aws.amazon.com/blogs/networking-and-content-delivery/introducing-ipv6-only-subnets-and-ec2-instances/

https://redd.it/r17xp6
@r_devops

Save your engineers' sleep: best practices for on-call processes

A solid on-call process is key to saving your engineers' sleep (and keeping them sane!)

Here's how we have optimized our on-call process at Ably:

☉ We treat alerts as code
☉ We use percentiles over averages
☉ We use playbooks to document each alert
☉ We use Prometheus Alertmanager
☉ We use PagerBeauty to show who is on-call
☉ We automate all our pages
☉ We use routine tests
☉ We have an incident management framework


This is full reasoning behind each one:

Save your engineers' sleep: best practices for on-call processes

https://redd.it/r17tv3
@r_devops

AWS Fargate: how to get the nginx container to read

We're using aws fargate, we have two containers, a PHP container and a nginx container. The files are mounted to the PHP container.

How to get the nginx container to read the static files (css/js/images) so that the pages are rendered correctly?


In docker-compose, I mount the volume on the nginx container and on the PHP container, but AWS fargate doesn't support docker compose

https://redd.it/r0yce8
@r_devops

Black Friday Deals For Developers 2021

Hey everyone,I hope you guys are doing fine.

I'm sharing here some awesome black Friday deals for developers 2021. I hope you find it helpful.

* [Themeselection - 25% Off Storewide](https://themeselection.com/)
* [Startup- 30% OFF](https://designmodo.com/startup/)
* [Vue School – Premium VueJS Tutorials – 40% OFF](https://vueschool.io/sales/blackfriday?)
* [Pluralsight - 40%](https://www.pluralsight.com/)
* [Mockplus – Product Design Platform – Up To 72% OFF](https://www.mockplus.com/buy/black-friday?)
* [Laracast – 50% Off](https://laracasts.com/)
* [75% OFF on DataCamp Annual Plan](https://www.datacamp.com/)
* [Udemy Black Friday Deals 2021 Up to 85%](https://www.udemy.com/)
* [Zyro – Website Builder – up to 86% OFF](https://zyro.com/in)

Thanks.

https://redd.it/r1vhmz
@r_devops

Help with circleci persist_to_workspace?

I'm running a circleci pipeline with a plan and apply step for terraform, and it is my first time using `persist_to_workspace` functionality so that i can cache the working directory, and use the outputted terraform plan file as an input on the apply step, not to mention it doesnt have to reclone the repo, or re initialise terraform and download all the plugins again. However the `persist_to_workspace` is not.... persisting to workspace.... I am following this example in the terraform documentation: [https://learn.hashicorp.com/tutorials/terraform/circle-ci](https://learn.hashicorp.com/tutorials/terraform/circle-ci). When running the apply job with SSH enabled, and SSHing onto the worker instance, there is nothing there in the working directory, nothing has been "persisted".

Below is an snippet from the pipeline, can anybody tell me what im doing wrong? Please dont tell me that I am missing a huge portion of the pipeline, I have simply just pasted what is relavent to the question, there is of course much more to the pipeline than this hahahah.

jobs:
plan-staging:
executor: terraform
steps:
- checkout
- terraform-plan:
aws_access_key_id: '${AccessKey}'
aws_secret_key: '${SecretKey}'
tf_var_file: 'staging/staging.tfvars'
tf_backend_config: 'staging/backend.hcl'
tf_cloud_token: '${TF_CLOUD_TOKEN}'
environment: 'staging'
- persist_to_workspace:
root: .
paths:
- .

apply-staging:
executor: terraform
steps:
- attach_workspace:
at: .
- terraform-apply:
aws_access_key_id: '${AccessKey}'
aws_secret_key: '${SecretKey}'
tf_cloud_token: '${TF_CLOUD_TOKEN}'
environment: 'staging'

https://redd.it/r1xg6g
@r_devops

Nginx rewrite requests in Nginx without extension and not ending in / to add .html

I need help to implement something like this from express static options.

Sets file extension fallbacks: If a file is not found, search for files with the specified extensions and serve the first one found. Example:

'html', 'htm'

https://redd.it/r1x4od
@r_devops

those that take on contract roles outside of your full time employment, how do you do it?

Lets assume one is fully aware of the time management issues that might arise from doing something like this.

What advise or cautionary tales can you give for someone looking to take on short term contract roles on the side while working their full time job?

bonus points, if you are canadian (to understand laws if any)

https://redd.it/r252sq
@r_devops

What are DevOps issues that nobody talks about?

Hey guys, what are some develop issues that no one seems to talk about? These can be technical issues or just general business issues. And why is no one talking about these issues? I'm doing some user research on DevOps and trying to get a better understanding of the field. Thanks!

https://redd.it/r26xsu
@r_devops

WTF is up with Nginx Plus?

So I’m interested in improving our current Nginx OS + ModSecurity setup with Nginx Plus, mostly for better upstream proxy support.

After getting frustrated with the AWS marketplace I contacted F5 sales… For a very minimal 4 instance setup (which could easily handle the request but I’d prefer more just to sleep better at night) split over test/prod, with their own build of ModSecurity (which apparently you need to pay extra for) they are talking US$40-50k p.a.

Honestly WTF… I can work around the upstream issues so it’s really just that Plus would be more convenient with maybe some neat features.

I’m just a little confused by their positioning. Does anyone actually use it and thinks it’s worth the price? I’d be happy to pay, idk, an order of magnitude less than they’re asking…and that seems like a good deal for them given my use case…

Edit: to clarify, I find it odd that the only pricing tiers are free or enterprise…

https://redd.it/r2hn6k
@r_devops

Loki performance tuning

Hi

We have deployed Loki distributed (2.4.1) to Redhat Openshift on AWS. here is the current components as well as their replica and resources:

* Ingester: 3 pods, 600m CPU, 5Gi ram, and 10Gi storage each
* distributor: 1 pod, 200m CPU, 700Mi ram
* Querier: 2 pods, 1500m CPU, 5Gi ram, and 10Gi storage each
* Query Frontend: 3 pods, 200m CPU, 500Mi RAM
* Gateway: 1 pod, 200m CPU, 100Mi RAM
* Compactor: 1 pod, 200m CPU, 7Gi RAM, and 40Gi storage
* Index gateway: 1 pod, 200m CPU, 1Gi RAM, and 20Gi storage

We also have Memcached (chunks, frontend, index queries, index writes)

Also here is the config file:

^(auth\_enabled: false)
^(server:)
^(http\_listen\_port: 3100)
^(log\_level: debug)
^(distributor:)
^(ring:)
^(kvstore:)
^(store: memberlist)
^(memberlist:)
^(join\_members:)
^(- loki-distributed-memberlist)
^(ingester:)
^(lifecycler:)
^(ring:)
^(kvstore:)
^(store: memberlist)
^(replication\_factor: 1)
^(chunk\_idle\_period: 30m)
^(chunk\_block\_size: 262144)
^(chunk\_encoding: snappy)
^(chunk\_retain\_period: 1m)
^(max\_transfer\_retries: 0)
^(chunk\_target\_size: 2500000)
^(wal:)
^(dir: /var/loki/wal)
^(limits\_config:)
^(enforce\_metric\_name: false)
^(reject\_old\_samples: true)
^(reject\_old\_samples\_max\_age: 168h)
^(max\_cache\_freshness\_per\_query: 10m)
^(max\_global\_streams\_per\_user: 5000000)
^(schema\_config:)
^(configs:)
^(- from: 2021-08-08)
^(store: boltdb-shipper)
^(object\_store: s3)
^(schema: v11)
^(index:)
^(prefix: index\_)
^(period: 24h)
^(storage\_config:)
^(boltdb\_shipper:)
^(shared\_store: s3)
^(active\_index\_directory: /var/loki/index)
^(cache\_location: /var/loki/boltdb-cache)
^(cache\_ttl: 168h)
^(index\_gateway\_client:)
^(server\_address: dns:///loki-distributed-index-gateway:9095)
^(index\_queries\_cache\_config:)
^(memcached:)
^(batch\_size: 100)
^(parallelism: 100)
^(memcached\_client:)
^(consistent\_hash: true)
^(host: loki-distributed-memcached-index-queries)
^(service: http)
^(timeout: 5s)
^(aws:)
^(s3: s3://aaa)
^(s3forcepathstyle: true)


^(chunk\_store\_config:)
^(max\_look\_back\_period: 0s)
^(chunk\_cache\_config:)
^(memcached:)
^(batch\_size: 100)
^(parallelism: 100)
^(memcached\_client:)
^(consistent\_hash: true)
^(host: loki-distributed-memcached-chunks)
^(service: http)
^(timeout: 5s)
^(write\_dedupe\_cache\_config:)
^(memcached:)
^(batch\_size: 100)
^(parallelism: 100)
^(memcached\_client:)
^(consistent\_hash: true)
^(host: loki-distributed-memcached-index-writes)
^(service: http)
^(timeout: 5s)
^(querier:)
^(max\_concurrent: 10)
^(query\_timeout: 3m)
^(engine:)
^(timeout: 5m)
^(query\_ingesters\_within: 1h)
^(query\_scheduler:)
^(max\_outstanding\_requests\_per\_tenant: 1000)


^(table\_manager:)
^(retention\_deletes\_enabled: false)
^(retention\_period: 0s)
^(query\_range:)
^(align\_queries\_with\_step: true)
^(max\_retries: 5)
^(split\_queries\_by\_interval: 15m)
^(cache\_results: true)
^(parallelise\_shardable\_queries: true)
^(results\_cache:)
^(cache:)
^(enable\_fifocache: true)
^(fifocache:)
^(max\_size\_items: 1024)
^(validity: 24h)
^(memcached\_client:)
^(consistent\_hash: true)
^(host: loki-distributed-memcached-frontend)
^(max\_idle\_conns: 16)
^(service: http)
^(timeout: 5s)
^(update\_interval: 1m)
^(frontend\_worker:)
^(frontend\_address: loki-distributed-query-frontend:9095)
^(grpc\_client\_config:)
^(max\_send\_msg\_size: 33554434)
^(grpc\_compression: snappy)
^(frontend:)
^(max\_outstanding\_per\_tenant: 1000)
^(log\_queries\_longer\_than: 5s)
^(compress\_responses: true)
^(tail\_proxy\_url: https://loki-distributed-querier:3100)
^(compactor:)
^(working\_directory: /var/loki/compactor)
^(shared\_store: s3)
^(compaction\_interval: 5m)
^(retention\_enabled: true)

^(retention\_delete\_delay: 2h)
^(retention\_delete\_worker\_count: 150)
^(ruler:)
^(storage:)
^(type: local)
^(local:)
^(directory: /etc/loki/rules)
^(ring:)
^(kvstore:)
^(store: memberlist)
^(rule\_path: /tmp/loki/scratch)
^(alertmanager\_url: https://alertmanager.xx)
^(external\_url: https://alertmanager.xx)

The main problem is, Loki is either too slow or it fails to show logs and labels, I tried to curl `/loki/api/v1/label` index and I take timeouts after 5 minutes. The only relevant logs I find are slow query and context canceled on the query frontend and context canceled on query nodes.

Also, when I check the metrics, there are no lack of resource and all components are working with no problem

Our cluster is not currently under load and not many people are using it

https://redd.it/r2rfii
@r_devops

How do you guys deal with SSL certificates?

When I discovered letsencrypt, I thought I had found a miracle solution !

But in reality, it’s as much of a pain as everything else.

How do you guys deal with creating, storing and regularly renewing certificates?

https://redd.it/r2hqey
@r_devops

Minecraft based DevOps proecjt

Hi there,

I want to increase my DevOps knowledge. Can anyone recommend a project related to Minecraft?

At work, I am expected to learn Terraform, Kubernetes and AWS (and some other stuff). I'm not sure where to start. I've never built a minecraft server.

​

I want to host and manage a java minecraft server. Any advice? Anyone know what I should learn to be able to do this. I have purchased and will work through terraform and also a kubernetes course on Udemy.

https://redd.it/r2pgm6
@r_devops

Starting my first proper DevOps job from Monday. Some questions about the culture and mentality aspects.

Hi there,

I have been working on the pure Ops side with Azure, SQL Server and a bit of Azure DevOps and Python for the past 4 years and I have landed a proper DevOps role now. I am set to start the new gig from this monday. The new role will have extensive involvement with AWS, GCP, Gitlab, Jenkins etc in addition to whatever I already know.

I am not worried about learning all the new tech but a bit confused on how to make a strong start to the new career. The team I am going into will be having experienced DevOps engineers with a lot of them having extensive dev experience prior to that. I am coming from an Ops background and is worried about not fitting in quickly from the start.

Are there any rules of thumb, or unwritten rules or mentality points that I can read on so that I understand the "art of DevOps" ?


Thanks in advance

https://redd.it/r2sgzk
@r_devops

Who has ever set up Artifactory as a docker registry?

I really need help with ways to create the proper cert to be able to login via command line using docker or podman. I’ve setup Nginx and able to get to the registry via my browser over https.

But every time I try to docker login, I keep getting: “x509 certificate is not valid for any names, but wanted to match etc etc”

https://redd.it/r310y3
@r_devops

As a devops engineer how your daily responsibilities differ from a pure sysadmin (windows or linux) jobs ?

1). As a devops engineer how your daily responsibilities differ from a pure sysadmin (windows or linux) jobs ?

2). Do you feel that with the Cloud the old day of managing everything in a datacenter or on premise by hand or long time gone or we will always need skill to install os, patch, administer stuffs because there is a limit to automation or CICD pipeline ? Will the old sysadmin jobs still exist or they will all need to program/script Azure Aws API with others tools like Jenkinds, Terraforms etc etc

https://redd.it/r335ul
@r_devops

What is the easiest cloud server provider to get started with?

I am learning more about devops and I am wondering what is the easiest to use and get started with?

How would you rank the cloud providers in terms of ease of use?

I started looking into AWS, but it doesn't feel as easy, mostly because it has lots of options and places where I could miss or forget. Is there anything that is easy and relatively cheap to get started with? Heroku is a bit easier but they charge a lot for their services.

https://redd.it/r38f0k
@r_devops

Which tools did you use to design yours cloud architecture ?

Hi community, currently I use r/drawio to design our cloud architecture. Whereas recently a partner tell that we are using old tools to do it. So here we come. Which tools did you use to design a cloud infrastructure?

https://redd.it/r3exqp
@r_devops

Behind the scenes of the night our transformer shut down in our data center

One night in September, a power transformer shutdown in one of our Parisian data centers.

While we were writing this article, this situation happened again for the third time in ten years. Like the two other times, our two power backups ensured the power lineup worked while our team rallied to bring the situation back to normal. Read on to find out what happened during this tense night.

​

We equip all of our data centers with a Scaleway-made building management system tool called SiMA. Thanks to this tool, we can monitor and analyze hundreds of thousands of real-time data points from our equipment. This allows us to have a complete overview of our infrastructure at all times, and to be able to optimize it to be as close as possible to customers’ demands.

We build our software and hardware to monitor our equipment because manufacturers’ products do not come equipped with the technical level we require.

It is common to see building management system tools exceed one million euros in our business.

So, we built our own and integrated it as an internal chatbot. Thanks to SiMA, we started receiving notifications at 05:09 AM, alerting us that one of our power lineups was no longer being supplied by the grid. Our technicians immediately checked the programmable logic controller and confirmed what we feared: SiMA was right, and we had a long night ahead of us. As soon as the failure occurred, the automatic switch to our generators had been made.

​

# First step: synchronize and assess the situation

​

We synched with our on-call engineers and board members, and notified our clients. At this point, we have an autonomy of five days on fuel oil, and 20 minutes on battery. The issue was likely caused by insufficient oil in the transformer itself. Our team quickly went to the site and found an oil leak by a transformer component called the Buchholz relay. This is a protection relay that acts as a sensor to monitor the temperature, oil level, and gas discharge of the transformer.

# Safe working conditions - isolate the high-tension unit

The fault with the Buchholz relay triggered the insufficiency, but luckily, we were only a few liters of oil short. We then started by creating a safe working environment by isolating the high-tension unit from the power transformer, while other members of the team searched for vegetable oil to stock up on - this proved to be quite a mission in itself as the incident occurred in the middle of the night.

We use vegetable oil instead of other types of oil to power our transformers, mainly for environmental and security reasons. The oil we use has a fire point of over 300°C, which makes it barely flammable. It also is bio-sourced, easily biodegradable, and non-toxic. Unfortunately, so far, our experience with vegetable oil has been pretty bad.

The power lineup continued to be fed by its two electric generators, supervised by our engineers. The company that handles the maintenance came to the site, too, ready to assist us. Even with two electric generators, you can never be too cautious. The faulty Buchholz relay was dismantled and checked to diagnose and understand what went wrong, and learn from there. The new relay was calibrated and then installed.

# We have now been relying on our electric generators for eight hours.

if you made it this far, the rest is here: with images, video and all: https://blog.scaleway.com/behind-the-scenes-of-the-night-our-transformer-shut-down-in-our-data-center/

https://redd.it/r39ouv
@r_devops

Has anyone given interview for SRE in FANG ?

Share your interview experience for an SRE role in FANG type org

https://redd.it/r3g8wz
@r_devops

GitHub is down (11/27)

All of GitHub appears to be down. https://www.githubstatus.com/

Congrats to anyone working on their side projects this holiday weekend in the USA lol

https://redd.it/r3mop1
@r_devops