Posts

Showing posts from January, 2017

How to analyze S3 logs and move to Local system.

Amazon web services move there web logs or any type of logs in S3 so in order to analyze them we need to parse them either via any third party application or download them individually and work on it.
Following script will help us to parse S3 files directly and create single file from it.
[root@ip-10-0-1-220 ravi]# cat s3logs.sh #!/bin/bash #By Ravi Gadgil #Script to get latest logs from S3 and sync to System log. echo -e "\n------------------------------\n`date`\n-----------------------------" s3location=$(echo -e "s3://teamie-logs/S3Logs") echo -e "The S3 Bucket location is : $s3location" #To check if its first time sync if [[ ! -f /tmp/lastfile.txt ]]; then #To get last 200 files names sorted by date. aws s3 ls $s3location/ | tail -n 200 | awk '{print $4}' > /tmp/filesync.txt for i in `cat /tmp/filesync.txt`; do echo "Following log file is being synced : $i " aws s3 cp $s3location/$i - >> /var/log/s3hits.log done #To …