• Skip to main content

Uly.me

cloud engineer

  • Home
  • About
  • Archives

Archives for February 2022

Python Boto3 S3 List Objects

February 27, 2022

How to list S3 objects using Python Boto3

import boto3
from sys import argv
bucket=argv[1]
s3 = boto3.resource('s3')
mybucket = s3.Bucket(bucket)
def list_buckets():
    print("List of Buckets: ")
    for bucket in s3.buckets.all():
        print(bucket.name)
def list_files():
    print("Buckets: The objects in the \"" + bucket + "\" buckets are ... ")
    for files in mybucket.objects.all():
        print(files.key)
def main():
    #list_buckets()
    list_files()
if __name__ == "__main__":
    main()

import boto3 from sys import argv bucket=argv[1] s3 = boto3.resource('s3') mybucket = s3.Bucket(bucket) def list_buckets(): print("List of Buckets: ") for bucket in s3.buckets.all(): print(bucket.name) def list_files(): print("Buckets: The objects in the \"" + bucket + "\" buckets are ... ") for files in mybucket.objects.all(): print(files.key) def main(): #list_buckets() list_files() if __name__ == "__main__": main()

Running command

$ python test.py bucket-name    # with python
$ ./test.py bucket-name         # without python

$ python test.py bucket-name # with python $ ./test.py bucket-name # without python

Results

Buckets: The objects in the "bucket-name" are ....
abc.txt
def.txt

Buckets: The objects in the "bucket-name" are .... abc.txt def.txt

Filed Under: Linux Tagged With: boto3, list, objects, python, s3

Python Virtual Environment

February 27, 2022

How to create a Python virtual environment.

On Windows

$ pip install venv
$ python3 -m venv ~/python-test

$ pip install venv $ python3 -m venv ~/python-test

On Ubuntu

$ sudo apt install python3.8-venv
$ python3 -m venv ~/python-test

$ sudo apt install python3.8-venv $ python3 -m venv ~/python-test

On Mac OS

$ brew install virtualenv
$ virtualenv python-test

$ brew install virtualenv $ virtualenv python-test

Activate

$ cd ~/python-test
$ source bin/activate

$ cd ~/python-test $ source bin/activate

Deactivate

$ deactivate

$ deactivate

Filed Under: Linux Tagged With: activate, deactivate, environment, python, virtual

Bash Reformat Numbers

February 27, 2022

How to reformat numbers in Bash.

$ n=1023555
$ m=$(numfmt --to=si n)
$ echo $m
1.1M

$ n=1023555 $ m=$(numfmt --to=si n) $ echo $m 1.1M

Other options

--to=none     # no scaling. Output: 1023555
--to=si       # International System of Units (SI) standard. Output 1.1M
--to=iec      # International Electrotechnical Commission (IEC) standard. Output: 1.1Mi 
--to=iec-i    # International Electrotechnical Commission (IEC) standard. Output: 1.1M
--to=auto     # ‘auto’ can only be used with --from.

--to=none # no scaling. Output: 1023555 --to=si # International System of Units (SI) standard. Output 1.1M --to=iec # International Electrotechnical Commission (IEC) standard. Output: 1.1Mi --to=iec-i # International Electrotechnical Commission (IEC) standard. Output: 1.1M --to=auto # ‘auto’ can only be used with --from.

fmtnum

Filed Under: Linux Tagged With: bash, convert, fmtnum, format, numbers

GCP Monitor Service Account

February 24, 2022

You can monitor the success or failure of a service account in GCP.

gcloud policy-intelligence query-activity \
--activity-type=ACTIVITY_TYPE \ 
--project=PROJECT_ID \
--limit=LIMIT

gcloud policy-intelligence query-activity \ --activity-type=ACTIVITY_TYPE \ --project=PROJECT_ID \ --limit=LIMIT

The two options you can use for ACTIVITY_TYPE are:

  • serviceAccountKeyLastAuthentication
  • serviceAccountLastAuthentication

Filed Under: Cloud Tagged With: gcp, keys, monitor, service account

GCP Windows Password Reset

February 14, 2022

Instead of the GCP Console, you can reset Windows Server password via gcloud.

gcloud compute reset-windows-password server-name \
--zone us-central1-a \
--project project-id

gcloud compute reset-windows-password server-name \ --zone us-central1-a \ --project project-id

Username and password will generated.

Filed Under: Cloud Tagged With: gcloud, gcp, password, reset, windows

Google SDK SSH Mac Terminal

February 14, 2022

I’m having trouble logging in using Google SDK Compute SSH on a Mac Terminal.

Here’s the fix.

gcloud compute ssh USERNAME@SERVER --zone ZONE --project PROJECTID --internal-ip 2>&1

gcloud compute ssh USERNAME@SERVER --zone ZONE --project PROJECTID --internal-ip 2>&1

There was an issue with a redirect to another shell.

Filed Under: Cloud, Linux Tagged With: compute, gcp, mac, redirect, sdk, ssh, terminal

AWS SDK Load Config

February 14, 2022

Occasionally I was getting this random error when running Terraform.

╷
│ Error: error configuring Terraform AWS Provider: no valid credential sources for Terraform AWS Provider found.
│ 
│ Please see https://registry.terraform.io/providers/hashicorp/aws
│ for more information about providing credentials.
│ 
│ Error: RequestError: send request failed
│ caused by: Post "https://sts.amazonaws.com/": read tcp xx.xx.xx.xx:59422->xx.xx.xx.xx:443: read: connection reset by peer
│ 
│ 
│   with provider["registry.terraform.io/hashicorp/aws"],
│   on main.tf line 10, in provider "aws":
│   10: provider "aws" {

╷ │ Error: error configuring Terraform AWS Provider: no valid credential sources for Terraform AWS Provider found. │ │ Please see https://registry.terraform.io/providers/hashicorp/aws │ for more information about providing credentials. │ │ Error: RequestError: send request failed │ caused by: Post "https://sts.amazonaws.com/": read tcp xx.xx.xx.xx:59422->xx.xx.xx.xx:443: read: connection reset by peer │ │ │ with provider["registry.terraform.io/hashicorp/aws"], │ on main.tf line 10, in provider "aws": │ 10: provider "aws" {

Here’s the fix. Place this in your ~/.bash_profile.

export AWS_SDK_LOAD_CONFIG=1

export AWS_SDK_LOAD_CONFIG=1

This forces Terraform to use both config and credentials file.

Filed Under: Linux Tagged With: aws, awscli, bash_profile, cli, config, credentials, sdk, terraform

Saml2aws

February 14, 2022

Use Saml2Aws CLI as an alternative to SAML to AWS STS Key Conversion.

Install on Mac.

brew install saml2aws
saml2aws --version

brew install saml2aws saml2aws --version

Configure. Provide information.

saml2aws configure

saml2aws configure

It will create a ~/.saml2aws config file. Set session to 8 hours.

aws_session_duration    = 28800

aws_session_duration = 28800

Login.

saml2aws login

saml2aws login

After authentication and/or MFA, your ~/.aws/credentials will be updated.

Filed Under: Linux, Mac Tagged With: aws, integration, keys, saml, saml2aws, sessions, sts

  • Go to page 1
  • Go to page 2
  • Go to page 3
  • Go to Next Page »
  • Home
  • About
  • Archives

Copyright © 2023