Ship CloudWatch Log to Grafana using lambda

I'm setting up a lambda function to push log from cloudwatch to grafana I then put a concurrency limit on my lambda so it won't throttle itself but I noticed that some of the logs are missing in grafana compared to cloudwatch. I was thinking about sending log in batch with sqs but cloudwatch subscription filter doesn't support it. How do you guys ship your cloudwatch log to lambda in batch?

https://redd.it/12ho81s
@r_devops

DevOps Learning Series Principles and a look at their impact.

Series index

Hey r/devops, continuing the Devops Learning series, this post focuses on the core principles of DevOps and the benefits they offer. Understanding these principles will help you embrace the DevOps mindset and successfully implement its practices in your organization.

## Devops Principles:

1. Collaboration: Encourage close cooperation and communication between development and operations teams, breaking down traditional silos, and fostering a shared understanding and ownership of the software delivery process.
2. Automation: Streamline and automate manual, repetitive tasks throughout the software development lifecycle, including testing, deployment, and infrastructure management, to reduce human error and increase efficiency.
3. Continuous Integration (CI): Regularly integrate code changes into a shared repository, triggering automated builds and tests to catch integration errors early and maintain a consistently releasable product.
4. Continuous Delivery (CD): Automate the release of code changes to production environments, ensuring rapid, reliable, and consistent deployments with minimal manual intervention.
5. Continuous Improvement: Embrace a culture of constant learning and adaptation by regularly assessing processes, tools, and practices, and making data-driven decisions to optimize performance and address issues.
6. Rapid Feedback: Implement robust monitoring and feedback mechanisms to quickly identify issues, learn from user behavior, and inform future development efforts, thereby ensuring continuous alignment with customer needs.
7. Infrastructure as Code (IaC): Manage and provision infrastructure resources using code and version control systems, enabling consistent, repeatable, and automated deployment of environments.
8. Security and Compliance: Integrate security best practices and regulatory requirements throughout the development lifecycle, promoting a proactive approach to risk mitigation and fostering a shared responsibility for security among team members (DevSecOps).
9. Flexibility and Adaptability: Embrace an iterative, experimental mindset that allows for rapid response to changing requirements, technology advancements, and market demands, fostering a resilient and agile organization.
10. Customer-Centricity: Prioritize the needs and expectations of end users, using their feedback to inform development decisions, and ensuring the continuous delivery of value-added features and improvements.

## Benefits

Let's take a closer look at the data supporting the effectiveness of DevOps principles. By exploring various studies and reports that shed light on the impact of DevOps, we'll further our understanding of the value these principles bring to organizations.

Deployment Frequency and Speed

One of the key indicators of a successful DevOps implementation is the frequency and speed of deployments. According to the 2021 State of DevOps Report by Puppet, high-performing DevOps organizations deploy software 200 times more frequently than low performers (1). These rapid deployment rates are enabled by effective automation, streamlined processes, and close collaboration between development and operations teams.

Additionally, the same report found that high-performing DevOps organizations have a lead time for changes (i.e., the time it takes for code changes to move from commit to production) that is 2,604 times faster than low-performing organizations (1). This acceleration highlights the efficiency gains that DevOps practices can bring to the software delivery process.

Reduced Change Failure Rates and Improved Recovery Times

DevOps principles not only accelerate the software delivery process but also improve the quality and reliability of releases. The 2021 State of DevOps Report revealed that high-performing organizations experience a 3 times lower

change failure rate compared to low performers (1). This improvement in reliability can be attributed to rigorous testing, automated deployment, and better collaboration between development and operations teams.

When failures do occur, DevOps practices contribute to faster recovery times. The same report found that high-performing organizations have a mean time to recovery (MTTR) that is 2,604 times faster than their low-performing counterparts (1). By implementing robust monitoring, swift incident response, and effective collaboration, DevOps teams can quickly identify and resolve issues, minimizing the impact on customers.

Enhanced Customer Satisfaction and Business Outcomes

Embracing DevOps principles not only improves technical performance but also leads to better business outcomes. A 2019 Accelerate State of DevOps Report by DORA and Google Cloud demonstrated a strong correlation between high-performing DevOps teams and superior business performance (2). High performers reported 1.7 times greater customer satisfaction, 1.7 times higher profitability, and 1.8 times higher market share compared to their peers (2).

In this post I communicated examples highlighting the value of adopting DevOps principles for both the technical and business aspects of organizations. By embracing collaboration, automation, continuous improvement, and customer-centricity, teams can deliver high-quality software rapidly and efficiently, leading to improved customer satisfaction and business performance.

As always, I encourage you to share your thoughts and experiences with DevOps principles and their impact on your organization in the comments below.

Sources and more reading:

(1) Puppet Labs. (2021). State of DevOps Report 2021. https://www.puppet.com/resources/report/state-of-devops-report/

(2) Forsgren, N., Humble, J., Kim, G., Brown, K., Kersten, M., Rivas, R., & Shipp, S. (2019). Accelerate State of DevOps Report 2019. DORA and Google Cloud. https://cloud.google.com/files/DevOps/state-of-devops-2019.pdf

https://redd.it/12hrjt4
@r_devops

Calling OpenAI API Service using Python

https://medium.com/techbeatly/calling-openai-api-service-using-python-9e1d51416e7a?sk=20568c9675cea21735d6b90ee0ee2d13

https://redd.it/12hnvfw
@r_devops

Need some help with terraformer

Hey All,

I am very new to this terraform world. I'm working on importing previously built hand-built AWS infrastructure into TerraForm.

I was using the terraform import function, however, it was time-consuming and difficult to use. I stumbled upon Terraformer today which is supposed to automate the task for you. I keep getting the following error message when performing the import. I was hoping someone with more experience than me could help me out.

​

Thanks!

​

​

Andrews-iMac:Import acaplice$ terraformer import aws -r vpc --regions=us-east-1 -O hcl --profile=ProfileName

2023/04/10 16:44:45 aws importing region us-east-1

2023/04/10 16:44:45 aws importing... vpc

2023/04/10 16:44:50 aws error initializing resources in service vpc, err: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, exceeded maximum number of attempts, 3, request send failed, Get "https://169.254.169.254/latest/meta-data/iam/security-credentials/": dial tcp 169.254.169.254:80: connect: host is down

https://redd.it/12hvsjl
@r_devops

Tools for checking your code?

What code tools do you use for verifying code you wrote,? Sonarqube ?

https://redd.it/12hmmjt
@r_devops

Building in public: Cloud pricing calculators are super annoying - so here is one based on natural language

For now I've only implemented AWS services, hence it's in alpha. But I'd be curious to hear what the most common pricing prompts are for you - and whether you mostly need estimates (only set the most important parameters on calculator.aws) or exact prices.

​

Also, how often do you look up cloud prices? Which ones do you look up? Let me know in the comments.

​

https://go.kubiya.ai/cloud-pricing (a video demo)

https://redd.it/12hz7wy
@r_devops

Platform Engineering for Hashicorp Vault?

We're setting up HCP Vault (the SaaS) to be a platform for other dev groups to use for their secrets. My team will be managing it, so I'm looking to templatize/standardize as much as possible.

I've read over this guide Onboarding Applications to Vault Using Terraform: A Practical Guide - https://www.hashicorp.com/blog/onboarding-applications-to-vault-using-terraform-a-practical-guide \- which is a very baby-steps way of structuring the terraform. Does anyone know of any guides with more complex terraform? I'm looking for other ideas for how different teams manage this, and not seeing a lot in public repos, maybe I'm missing it, or maybe anyone doing this work is keeping it private.

https://redd.it/12i17s4
@r_devops

Is it reasonable for a company to give you academic/behavior assessments before you’ve even gotten a phone screener?

I’ve been looking for jobs lately and have come across a few instances where a company wants me to take a Wonderlic type logic test or a 200 question personality assessment as part of the standard application. Sometimes multiple. And they’ll quote an expected 1-2 hours of time needed to complete them. Is this now normal? I personally think it’s ridiculous to ask for 1-2 hours of someone’s time just to submit a resume, especially because there’s no guarantee after spending that time I’ll even get the 10 minute phone screener. If I’m already in the running for the position and we’ve at least talked before and decided to continue, I’d be much more willing, but having to do them just to get a resume on a desk seems like an unreasonable ask.

Im asking because I’m not sure if I’m out of touch for how applications go these days or if this is a bit much for just an application

https://redd.it/12hmhuf
@r_devops

GitHub Secrets: Is it safe to store an environment secret to a file from within a GitHub Action?

In other words, is it safe to have something like this in a .yml file:

# Add API key
- name: Add API key secret to file
env:
API_KEY_SECRET: ${{ secrets.API_KEY_SECRET }}
run: echo "$API_KEY_SECRET" > ~/work/MyProject/MyProject/secrets/api_key.txt

To clarify, the secret is not pushed to the repository, initially it only exists as an environment secret. I'm not sure if the .txt file created above ends up somewhere unsecure. The reason I ask is because if this is secure, then I can have a convenient location for programs to access secrets stored in files both on my machine (either .gitignore or outside the local repo) and when running on the server, without storing secrets in the repository.

https://redd.it/12i56ye
@r_devops

DevOps/DevSecOps hero to zero

Hi guys,can you share your thoughts and a roadmap for being devops/devsecops, i am currently working as SOC Analyst and learning pentesting,i want to add DevOps to my skill and become DevSecOps.
Maybe in future i want to change my job to DevOps
Last question,what are the chances find remote work as a DevOps

https://redd.it/12i9csf
@r_devops

How much building a big following on the dev community can help someone get better job opportunities?

Im not sure if this is the best place for asking it but it’s something that makes me really curious! Especially concerning frontend professionals, since i see so many people apparently doing a big effort towards getting a bigger following on the community while in other hand a lot of other devs apparently aren’t active on social medias at all

https://redd.it/12ic2et
@r_devops

Open Policy Agent

I recently came across OPA. Currently exploring courses from Styra. I wanted to know if learning it is worth for a DevOps engineer? Is it in demand? Does this have any future In DevOps market?

https://redd.it/12icbhu
@r_devops

How much would you earn as a Senior Consultant in DevOps in AWS Poland?

Can anyone tell me the salary range of a Sr DevOps Consultant in Poland?

https://redd.it/12idrgi
@r_devops

8 Talks That You Can't Miss On KubeCon EU 2023!

https://youtu.be/eYM5IIQzK3k

https://redd.it/12ifj6q
@r_devops

Is Udemy certificates worth anything?

I know usually the answer to this is no. I'd appreciate it if you could hear my situation first, I am a Senior Sysadmin with 7 years of experience. I am looking to switch to DevOps. I have a little bit of DevOps exposure. But I do not have any DevOps experience or qualifications.

I just completed a 50 hours DevOps course on Udemy. It's pretty thorough and hands-on. Should I include it in my resume? I know it's just a certificate of completion. But isn't something worth nothing since I have no other DevOps certifications yet? (such as AWS or Kubernetes - which I am studying for but haven't completed yet).

https://redd.it/12ifdal
@r_devops

What would you consider a high pressure DevOps role?

For frame of reference:

Working 45-50 hours per week.
No breaks, expected to work through lunch.
On call rotation which is weekly where you may receive many overnight calls.
Shift work, needing to have early AM start or evening hours. No overnight hours yet.

Expected response times to complaints via chat or email is minutes. Emails or customer requests should be answered within the hour.

Is this normal for the industry or higher than average levels of demand/stress?

https://redd.it/12iki4b
@r_devops

KubeCon -- Rooftop Happy Hour

Hello everyone,
As KubeCon approaches next week I’d like to invite all attendees to join us for a lovely evening in Amsterdam. The FluxNinja team is hosting a happy hour event on Tuesday, April 18th.

Location & Time: Canvas, Amsterdam at 6 pm

You're welcome to come, enjoy drinks, and capture Amsterdam from a rooftop bar!

Here is the link for signing up for the waitlist: https://partiful.com/e/Af6VA6NzXIPxpv2ku6vk

https://redd.it/12ilykk
@r_devops

Learning path

I’ve been a data scientist for about 2 years now and only worked mostly with raw data and modeling, but now at my current role I’m being asked to run also the deployment part with techs like docker, Jenkins, Argo etc and recently I’ve been asked to learn about Prometheus and Grafana.

I didn’t have much time to even dig deep into docker and I feel like I can’t even gasp the basic knowledge of what each tech piece does, its purpose and in which part of the deployment tray it fits.

So, I wanted to know if you guys could start over how would you gasp the devops environment and it’d be pure gold if you could add a free courses as a recommendation

https://redd.it/12iqsqy
@r_devops

Needing to work with developers to fix something feels like asking them to do sidequests.

How do you guys handle tickets that require time from developers? I have a couple of Jira tickets that require either knowledge from specific developers to help fix something they created or require them to modify something in their code.


The problem I'm facing is that I need to chase them down to get them to help me and there's really no consequence if they decide not to. Boss doesn't seem to think this is an issue since they have their own tickets that they're working on, but my Blocked tickets are piling up and they have my name attached to them.

https://redd.it/12isaka
@r_devops

bash/shell scripting

when and why was last time you used bash or shell scripting ?

https://redd.it/12iptwr
@r_devops