How to analyze S3 logs and move to Local system.

Amazon web services move there web logs or any type of logs in S3 so in order to analyze them we need to parse them either via any third party application or download them individually and work on it.
Following script will help us to parse S3 files directly and create single file from it.
[root@ip-10-0-1-220 ravi]# cat s3logs.sh 
#!/bin/bash
#By Ravi Gadgil
#Script to get latest logs from S3 and sync to System log.

echo -e "\n------------------------------\n`date`\n-----------------------------"

s3location=$(echo -e "s3://teamie-logs/S3Logs")
echo -e "The S3 Bucket location is : $s3location"

#To check if its first time sync
if [[ ! -f /tmp/lastfile.txt ]];
then

#To get last 200 files names sorted by date.
aws s3 ls $s3location/ | tail -n 200 | awk '{print $4}' > /tmp/filesync.txt

for i in `cat /tmp/filesync.txt`; do
echo "Following log file is being synced : $i "
aws s3 cp $s3location/$i -  >> /var/log/s3hits.log
done

#To record last file which was synced.
tail -n 1 /tmp/filesync.txt > /tmp/lastfile.txt

else

#To get files names sorted by date.
aws s3 ls $s3location/ | awk '{print $4}' > /tmp/filesync.txt

#To get last name file
lastfile=$(cat /tmp/lastfile.txt)

echo -e "Last synced file was : $lastfile"

#To get list of files need to sync after last sync
sed -e "1,/$lastfile/d" /tmp/filesync.txt > /tmp/newfiles.txt

if [[ -s /tmp/newfiles.txt ]];
then
 

 for i in `cat /tmp/newfiles.txt`; do
 echo "Following log file is being synced : $i "
 aws s3 cp $s3location/$i - >> /var/log/s3hits.log
 done

 #To record last file which was synced.
 tail -n 1 /tmp/newfiles.txt > /tmp/lastfile.txt

 else
 echo -e "No Files find to be synced"

 fi
 fi

Script will initially parse 200 files and add there data in single file in location /var/log/s3hits.log. If you want to parse all the files remove tail -n 200 it will take all the files till now in mentioned S3 bucket.
After the initial sync it will only take the new files after the previous sync so place this script in cron according to your need.

Comments

Popular posts from this blog

Script to create daily AMI of AWS server and delete AMI older than n no of days..

How to delete multiple user in linux

How to create users from ansible with public key and password.