Handling botocore.exceptions.ClientError: How to handle InvalidObjectState?

I just picked up python as my go-to scripting language and I am trying to figure how to do proper error handling with boto3.

I am trying to handle an exception that occurs when the object being called is on GLACIER. Instead of erroring out, I just want to print a message and make the loop keep going.
I can handle other botocore exceptions (like `if botocore.exceptions.ClientError.response['Error']['Code'] != "404"` for Object Not Found -- which works as expected.

I could not find an error code for InvalidObjectState. I tried these two approaches below without success:


```python
...
try:
storage_class = s3.Object(BUCKET_NAME, KEY).storage_class
except:
storage_class == "GLACIER"
print("STORAGE CLASS IS: ", storage_class)
pass
...
```

```python
...
try:
obj = s3.Object(BUCKET_NAME, KEY).load()
except botocore.exceptions.ClientError as e:
if e.response['Error']['Code'] == "InvalidObjectState":
pass
...
```

The problem is that it always errors out with the message:
*botocore.exceptions.ClientError: An error occurred (InvalidObjectState) when calling the GetObject operation: The operation is not valid for the object's storage class*

https://redd.it/e3kstl
@r_devops

Is a college degree needed for this type of career?



https://redd.it/e3lb5b
@r_devops

Workflow with Docker, Docker Compose, AWS ECR and Ansible

Trying to figure out the best workflow here. I inherited all that stuff a few months ago and I'm still learning.

For the past years until today:

1. 5 related services, each one has its own docker-compose.yml defining a single service
2. I deploy by: ssh on a server > "git pull" x 5 > "docker-compose up --build -d" x5 > repeat on the next server

Edit: the servers are bare metals, they are not on AWS.

---

Now I understand a bit more what's the right way to use Docker:

1. I should build the 5 images once and push them in a registry (I'll use AWS ECR)
2. I should have a single docker-compose.yml defining 5 services based on the images pushed in the registry
3. I could use Ansible (which I already have to provision/configure the servers) and the [docker-compose module](https://docs.ansible.com/ansible/latest/modules/docker_compose_module.html) to deploy on the servers

Am I right so far? Any suggestions?

(once this works, I guess the next step could be to add a CI/CD to build the images and deploy the containers automatically)

---

I still have some doubts tho:

1. should I merge the 5 git repositories in a monorepo to create the single docker-compose.yml in the same repo?
2. should I keep 5 repositories and define the docker-compose.yml in my Ansible project instead?
3. if 1, how do I access the docker-compose.yml from my Ansible project?
1. Do I copy/paste it in the Ansible project? (ew! not DRY, 2 sources of truth, risk to forget to update one)
2. Do I have to clone the monorepo on the Ansible control node to access its docker-compose.yml? (meh!)
4. if 2, how do I run the services in dev? Do I have to apply the Ansible playbook on my own machine? (meh!)
5. should I create a new repository only for the docker-compose.yml? (meh! introduce one more dependency…)

**TL;DR: where do I define the docker-compose.yml to make this Docker/Ansible workflow works?**

Thanks!

https://redd.it/e3j3ko
@r_devops

Outsourced company, do we need to replicate their setup

Hi,

Don’t wanna get into an A vs B argument, but could someone with actual experience advise and clarify please.

I work for a large company, we’re a team of 2 developers and 1 data guy. All our work is coded on Visual Studio and held on Azure DevOps Repos. We’ve a pipeline that takes the repos and pushes to an EC2 instance via CodeDeploy, all in Azure DevOps.

Works perfectly, never had a days problem.

We’ve had to outsource a large project to an external company, very well respected, and their setup is Jira with BitBucket and CircleCI to a serverless setup.

Ultimately, we’ll go serverless rather than EC2.

However, our Tech manager wants us to stop using Azure DevOps and go the same way as our outsourced “for about a year” as he feels working together and when the time comes for us to take over the project, we’ll have lots of problems.

Now I’m no expert on this, but surely if our outsource company is using BitBucket, we can jump into their repository quite easily, and any changes we/they make will be shared between both companies with ease.

Likewise, while we don’t use CircleCI ourselves, we could easily add their pipeline config in Azure DevOps (even add CircleCI itself) and again replicate the pipeline (via a 10 line YML config)

Am I wayyyyyy off or can it be done?

Like I said, really don’t want to get into a Jira+BitBucket+CircleCI vs Azure DevOps argument, just under a firm belief that both companies can easily stick with what they have.

Thanks.

https://redd.it/e3kjww
@r_devops

DevOps Career Question

Hello Team : I am been having challenges to find a job in IBM AS400 System Administration and would like to upgrade my skills to new technologies such as DevOps or Cloud Engineer.

(1) Would you please suggest what would be the best path for me specially I am experienced in System Administration and familiar with computer operations and SDLC.

(2) Any comments or suggestions on Edurekha or Udemey online course site that provides certification in DevOps or Cloud ?

Thank you.

https://redd.it/e3j4c6
@r_devops

Is anyone here working in a full remote position as DevOps/SRE ? I have a few questions for you .


* what did you look for before joining like stacks , teams etc . ?
* can you list some of the cons ?
* Is your team international, or the market you work for is in different Timezone?

Anything else you want to share ?

https://redd.it/e3a087
@r_devops

Dockerifying a Flask App for ECS -> Nginx+Gunicorn+Flask in 1 container?

Long story short, I'm looking to verify my approach is correct/decent.

Since you don't deploy Flask with their deployment server that Flask ships with, you use nginx as a web server because of ('slow clients', 'better security', and a ton of other features). Behind that you have something that implements the WSGI spec, aka Gunicorn to feed a request through your Flask app.


I realize you can pull Nginx out and people run it on its own server, but I'm not looking to do that. I want to ask about bundling these all into a single docker image for use on ECS.


Am I perfectly fine in throwing together a shell script which includes the necessary commands+config to set these both up. Then RUN that on the Dockerfile and I'm off to the races? The only weird thing here is that kind of means checking that into source control because I was planning on updating my artifact story with the newest docker image if tests pass.

https://redd.it/e3riwk
@r_devops

How do you maintain consistency between your local development and

production/staging environments?

Currently having issues with getting consistency between our developers developing locally andproduction/staging envs. We want it so whatever they do locally, their changes can be replicated on staging/prd. How do you manage this issue? What are some of the workflows your org uses to enable consistency between envs?

https://redd.it/e306fz
@r_devops

Anyone here use IaC to rebuild their laptop e.g. once per week?

Just curious. I remember seeing years ago an internet cafe that rebuilt its machines on every single reboot via PXE.

With the age of cattle rather than pets, I'm wondering if anyone here rebuilds their personal system on a regular basis. Of course it would necessitate reinstalling apps and keeping data somewhere else and mounting it (in house or cloud) but would have many benefits.

https://redd.it/e2zh1y
@r_devops

SAST and DAST

Can we achieve sast and dast both using Checkmarx tool? I've found a lot for SAST using Checkmarx but not DAST.

https://redd.it/e2yw67
@r_devops

Understand Private Docker Registry with this video 🙌🏻

After watching this video ([https://youtu.be/vWSRWpOPHws](https://youtu.be/vWSRWpOPHws)) you will be able and understand:

⭐️ How to create a Private Repository on AWS ECR, login and push to it

⭐️ Differences of Public Docker Hub and Private Repository

⭐️ Image Naming Concept in Docker Registries

​

Check it out and let me know what you think 🤓

I appreciate any feedback and suggestions for what I should cover next. 🙏

​

Btw this video is part of a **complete Docker & Kubernetes video series**, which you can find here: [https://www.youtube.com/playlist?list=PLy7NrYWoggjwPggqtFsI\_zMAwvG0SqYCb](https://www.youtube.com/playlist?list=PLy7NrYWoggjwPggqtFsI_zMAwvG0SqYCb)

https://redd.it/e3xgx1
@r_devops

Switching to devops

Hello guys, in the past year and a half I've been learning microservices, docker, kubernetes, aws, and linux. I've been working as a QA engineer for the past 5 years and I want to switch to devops. The company i work for hesitates to move me to devops as there's nobody there to keep me in his shadow for a few months to ramp up. Other companies wouldn't take me because i don't have experience in production/development environment.

What should I do? Thank you

https://redd.it/e3zqzt
@r_devops

Best open-source/free Artifactory

I'm looking for some recommendations on how you guys are storing your artifacts for your environments. We're looking to implement a more modern "DevOpsy" solution in our stack.

Current method is storing release artifacts in a external HDD, so literally any suggestions are better.

Had a look at Apache Archiva but the website alone makes me think twice.

https://redd.it/e2w9if
@r_devops

How to replicate the workflow of Platform.sh

As an agency, our team has been using [Platform.sh](https://Platform.sh) as our dev/ staging environments (and sometimes production) but due to performance issues we're considering rolling our own solution instead. Basically with as few headaches as possible.

Our current setup connects our Bitbucket repos to platform, so when we push to a branch it will build (composer and npm etc.) based on a platform.yaml file, and create an environment for any activated branch with a url to preview it.

What would be the easiest way to set this up if doing it ourselves? Since we're using Bitbucket I've already used BB Pipelines to build, but the deployment/hosting part is where I get a bit stuck. Every article I read about Docker/Kubernetes hosting kinda blows my mind with jargon - is there a super easy and simple setup guide anywhere?

Keeping costs low is also a consideration, although as we will be hosting many apps/sites we don't expect it to be cheap.

As far as urls for previewing our apps - I've read that Traefik would be able to do this with a DNS wildcard pointing to Docker containers based on their label.

Any input on this would be highly appreciated!

https://redd.it/e2qx4l
@r_devops

Anybody know of a really good tool for documentation?

Preferably Linux, and simple to use.

https://redd.it/e410yb
@r_devops

Trying to create a structured learning path, need suggestions about Linux Academy

If my main goal is to become a really good “DevOps” / SRE, which would be more beneficial of a learning path on LA (taking into account the industry is headed towards the cloud):

1) Junior Linux System Administrator -> Junior DevOps Entry Level -> Linux System Administrator Mid Level - Linux System Admin Senior Level

2) Get RHCSA -> Junior DevOps Entry Level -> ???

Would love suggestions. I would also be doing this combined with trying to get a job for experience / side projects. I currently know Python at an intermediate level as well.

https://redd.it/e3yq3j
@r_devops

I want to teach someone DevOps principles and tools

Following from a lovely post on r/learnprogramming [here](https://www.reddit.com/r/learnprogramming/comments/dq9isz/i_want_to_teach_someone_programming/), and looking at the threads on this sub, I thought it might be a good idea to do something similar here.

Thanks to u/fakehillbillyaccent for the original post. I've borrowed generously from it.

**Why?**

I enjoy teaching, it gives me a lot of fulfillment. Following from the above thread, I have been working with one person to help them learn programming and it has been a blast.

**Who am I?**

I have been around the block and found myself gravitating towards programming in general, and DevOps in particular. I've designed oilfield equipment, managed LEAN processes for manufacturing, full stack development, DevOps, and now I run the tech team for a small start up. I've worked professionally and full time with Java, javascript, Python, AWS, Google Cloud, Kubernetes, Terraform, Gitlab CI, and others.

**What can I offer?**

I would like to take on one person to help you through any challenges you may be having. I would prefer to work with someone trying to get a foothold as a DevOps engineer (or anything else you want to call it). We can start with Kubernetes operators or with `bash` scripting or anything in between.

If you are interested, please send me a message or comment and we can talk more.

from u/fakehillbillyaccent

>Again, this is completely free, no strings attached. You want to stop? Stop. Want to keep going? Cool. This is not an ad for a paid service or a trial for a paid service. I just like doing it and don't have enough opportunities to express myself through teaching.

If there are more people on this sub like me who are willing and able to help, please comment.

https://redd.it/e45hft
@r_devops

Custom FrontEnd app to Chef Open Source server

Hi all

We use Chef 14.x for config management across 1000+ nodes with the Open Source server.

We use environment files to pin cookbooks and manage versions, but am considering Policyfiles.

The nodes come in many different flavours which means we have to maintain different environment files for groups of nodes. Each release has a time stamp env.json file so it’s getting messy to manage manually.

Also about 400 of the nodes have a unique value for one particular app configuration so at the moment have to create separate environment file for each of those nodes to specify attribute value.

Problem we have is no dB or UI to centrally manage workflows.
So I am thinking of writing a centralised custom app which would provide workflows for use cases to configure different flavours of nodes for the different teams involved. This way all the node’s custom attribute could be represented as UIs the users understand and all the environment files hidden behind the scenes.

Does Automate provide all this?
Anyone have any suggestions?

Regards

https://redd.it/e3t0xz
@r_devops

How to automate installing an ISO in a VM?

We're trying to develop CI/CD processes to automate testing and installing our builds. Ya know, as one does in DevOps.

The end result of our build process are ISOs that use Red Hat Kickstart to install RHEL + our software. Today we burn the ISOs to DVD and install them on bare metal hardware. I would like to some how install and test these ISOs on virtual machines as part of our CI process. Something like:

1. Push commit to GitLab.
2. GitLab CI process runs.
3. Generates an ISO.
4. Creates a half dozen VMs via Ansible/Vagrant/something else??
5. Installs ISO in each VM.
6. Runs a smoke test on each VM to verify that it boots and our software runs.

I'm not really sure how to do 4, 5, and 6. What tools should we be using? Welcome any tips or advice... even just names of automation tools to look into.

https://redd.it/e2isxt
@r_devops

What makes a good runbook?

Thought that this article I wrote about the attributes of a good runbook might be of interest:

[https://www.transposit.com/blog/2019.11.14-what-makes-a-good-runbook/](https://www.transposit.com/blog/2019.11.14-what-makes-a-good-runbook/)

In particular, I'd love to hear feedback on what I missed. I also want to acknowledge up front the tension between runbooks and automation--the more you automate, runbooks seem to get:

* more high level (becoming more of a guide)
* more esoteric (for dealing with edge cases)
* lower value (for dealing with situations that are not worth automating)

https://redd.it/e2np4q
@r_devops