the project is to create a data pipeline. This should be a simple project for someone who is familiar with aws.
The objective is to look at an S3 folder every hour. If there is a file in that folder copy it to another folder. Failures will send or log a notification.
Thank you.
Hi
I'll use AWS S3 SDK to create the program.
The program will be scheduled to run every hour via crontab.
I'm confident enough to finish it within 2-3 hours at max.
THanks
Joy
I can write a program for you in Java that will check an S3 folder every hour and copy any files in that folder to another folder. I have used the Amazon AWS API for Java for a couple of S3 projects here before, so I am familiar with it.
Let me know if you're interested. Hope to hear from you soon.
Regards,
Christopher (andrewscm)
Should be simple enough, get s3 bucket objects, store list, cron job, compare with stored list, copy content to another folder, sns if failed.
In what language you expect it to work? bash script, ruby, python, php or what script?