JQ Proccessor

The AWS CLI spits out a JSON output after each successful execution. If you need to grab the result, assign it to a variable, and use it for your subsequent scripts, you need some kind of JSON parser. You can use a tool like jq which will process or filter out the result for you. From jq’s website,

jq is a tool for processing JSON inputs, applying the given filter to its JSON text inputs and producing the filter’s results as JSON on standard output. The simplest filter is ., which is the identity filter, copying jq’s input to its output unmodified (except for formatting).

In this example, we will use the AWS CLI to give us a list of running EC2 instances. We will then dump the output into a file called output.json. We will then filter out the “InstanceId” by running it through cat and the jq processor. We will then assign the result to a variable called INSTANCE, and then finally use that variable to associate our instance to an Elastic IP Address.

aws ec2 describe-instances --filters Name=instance-state-name,Values=running > output.json
INSTANCEID=$(cat output.json | jq '.Reservations[].Instances[] | {InstanceId} | .InstanceId' --raw-output)
aws ec2 associate-address --instance-id $INSTANCEID --public-ip xxx.xxx.xxx.xxx

Filtering a nested JSON can be a bit tricky. In this particular case, we are using a filter you’ll find inside the single quote right after the jq command. To remove quotes from our result, I’m using –raw-output switch. Finally, I then associate our instance to an elastic public IP address.

jq is a very handy tool.

Install Boto3 AWS SDK for Python

Boto3 is the Amazon Web Services (AWS) SDK for Python, which allows Python developers to write software that makes use of Amazon services like S3 and EC2. Boto provides an easy to use, object-oriented API as well as low-level direct service access.

To install on Mac

sudo easy_install pip
sudo pip install —ignore-installed six boto3

You need to set up your credentials and config file to authenticate to AWS first. The files are:


After it’s installed and configured, try using Boto3 to fetch your AWS S3 buckets.

import boto3
s3 = boto3.resource('s3')
for bucket in s3.buckets.all():

Your S3 buckets will be listed after execution.

Boto3 has access to other AWS resources like EC2, meaning you can start and stop instances via Python and Boto3.

Install AWS CLI on Ubuntu

The advantage of running Amazon Linux on AWS is that you already have the AWS CLI (command line interface) loaded as part of the image. In addition, Amazon Linux has rolling updates. It means you don’t have to perform major upgrades as you go to from one version to another. It’s all part of the process when you perform periodical updates. However, if you’re running Ubuntu, it’s no slouch. It’s a popular distro which many administrators have chosen to use due to its simplicity and robust repository. So, if you’re running Ubuntu, you might want to install the AWS CLI to give you the tools you need similar to what Amazon Linux has. Here’s how to install AWS CLI on Ubuntu.

$ sudo apt install awscli

You will need to run the configuration before you can start using AWS CLI. See example below.

$ sudo aws configure
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: us-west-2
Default output format [None]: json

Obviously, you will need your own Access Key and Secret Key that you’ll need to generate from your AWS Console under IAM.

NextCloud Install

NextCloud is an open-source next-generation file sharing and file synching web-application that you can run and install on your own Linux server. It’s similar to Dropbox, the file-sharing cloud application. The big difference between Dropbox and NextCloud is that you get to run your own cloud storage on your own server. The following are instructions on how to install NextCloud on an Ubuntu Server 16.04 LTS server.

Run Updates First

$ sudo apt-get update
$ sudo apt-get upgrade -y
$ sudo apt-get dist-upgrade -y

Install Apache

$ sudo apt-get install apache2

Install MySQL Server

$ sudo apt-get install mysql-server mysql-client
$ sudo mysql_secure_installation

Install PHP

$ sudo apt-get install php libapache2-mod-php php-mbstring php-zip php-xml php-mysql php-gd
$ sudo apt-get install php-json php-curl php-intl php-mcrypt php-imagick php-dom unzip

Edit php.ini change the memory limit and upload and post file sizes to something bigger. Restart Apache.

$ sudo nano /etc/php/7.0/apache2/php.ini
memory_limit = 512M
upload_max_filesize = 500M
post_max_size = 500M
$ sudo service apache2 restart

Create Database and User. Flush privileges.

$ mysql -u root -p
mysql> create database nextcloud;
mysql> grant all privileges on nextcloud.* to 'ncuser'@'localhost' identified by 'password';
mysql> flush privileges;
mysql> exit;

Download and Install the latest Nextcloud. Give permissions to Apache.

$ cd /var/www/html
$ sudo wget https://download.nextcloud.com/server/releases/nextcloud-11.0.2.zip
$ sudo unzip nextcloud-11.0.2.zip
$ sudo chown -R www-data:www-data nextcloud

Setup Apache

$ sudo nano /etc/apache2/sites-available/nextcloud.conf
Alias / "/var/www/html/nextcloud/"
 Options +FollowSymlinks
 AllowOverride All
  Dav Off
 SetEnv HOME /var/www/html/nextcloud
 SetEnv HTTP_HOME /var/www/html/nextcloud

Enable Nextcloud. Disable default.

$ sudo a2ensite nextcloud.conf
$ sudo a2dissite 000-default.conf

Enable the following

$ sudo a2enmod rewrite
$ sudo a2enmod headers
$ sudo a2enmod env
$ sudo a2enmod dir
$ sudo a2enmod mime
$ sudo service apache2 restart

Open your browser and access your server via IP address or domain name to complete the NextCloud install.

You will be asked to enter a new username and password. Also enter your database credentials to complete the installation.

Working With AWS LightSail

Just a quick demo how to work with AWS LightSail. LightSail is AWS lightweight VPS at a fixed price point. You can install just the OS. You can choose either Amazon Linux or Ubuntu OS. If you prefer, you can install any of the several ready-made applications provided by AWS in conjunction with Bitnami. In this example, I installed Ubuntu and Apache just demonstrate how to easy it is to work with LightSail.