Find Jobs
Hire Freelancers

AWS API/S3/Lambda Developer Needed

$250-750 USD

Closed
Posted over 5 years ago

$250-750 USD

Paid on delivery
Established b2b company with local, COLO, cloud and virtualized services needs help on a special AWS project. We need a multi-step utility/development project to utilize a few AWS services as part of a collective “project”. What We Currently Do: -We have hundreds of data partners. They provide us API endpoint access to download updated values/data. - We invoke their API in a .NET solution. - Response data is saved locally. - We have to run multiple API GET’s, liken to looping through page=1, page=2 sometimes. - Requires a good bit of computing power and resources, especially now that we are scaling. What We Want: - We want to invoke one (1) AWS API. By invoking this one API Endpoint all further processing will be done in AWS and return a “job status” code or results/total successful API’s, etc. - We want to pass to AWS API a list of all API URL’s and/or path-variables and/or query string parameters. - This batch list of urls that need to be invoked will be sent somehow to AWS, such as in a HTTPWebRequest Body (JSON). - AWS wil have to “loop through” or “process” list of URL’s. I’m not sure if this is Lambada or Batch, there is no preference except light-weight and serverless. - After AWS API returns a response (which is always in JSON format), we want each response saved as a .json file in an AWS S3 Bucket. - A method to easily query/index S3 Bucket contents. Some buckets will have millions of files. We need to be able to quickly determine if a filename exists (CloudSearch?) Considerations: - You will have to create AWS API Gateway and simply the process for passing querystring parameters for each URL (example: zip = 24018, pagenum = 1) - You will have to create AWS S3 Bucket and consider permissions, access, etc. - You will have to create some sort of “job” or “batch” fuction that will tell AWS API to update each URL in the .json body. - You will have to provide a simple and extremely responsive/quick way to return S3 Bucket file search properties. - You will have to create a new IAM user just for this project. - You must lock down the IAM user to only these resources. - You must limit the region for the IAM user to have access to only US-East1. - The IAM user will have rights to invoke AWS API, create/manage/run Lambda Functions and/or AWS Batch Jobs and S3 Bucket Read/Write permissions for the buckets that only pertain to this project. Pseudo-Code Diagram - Attached is a basic visual diagram of the flow/goal of thisi project. This is not meant to be static nor correct. We need help designing the most efficient and cost effective way to accomplish the goals outlined in the diagram. - It is not mean to be “accurate”, rather an overview of what needs to happen and the net result of how to save the URL response data into S3 Buckets.
Project ID: 18116064

About the project

7 proposals
Remote project
Active 5 yrs ago

Looking to make some money?

Benefits of bidding on Freelancer

Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs
7 freelancers are bidding on average $684 USD for this job
User Avatar
amazon web services expert
$500 USD in 1 day
5.0 (32 reviews)
5.6
5.6
User Avatar
Hi there, Though there're several points we need to discuss in detail for the moment the requirements look quite clear and straighforward to implement. Apparently, the main operation is not compute bound but rather a network bound task. AWS Batch is specifically designed for compute heavy tasks like rendering, analytics, evolution algos. so it's definitely not something you need. AWS Lambda is a super scalable service. Every time a function is invoked a new instance is spawned. For your use case at each invocation a GET request will be sent to the specified url and the response will be saved in a json file on s3. Since that's a very lightweight operation I don't think there's any use treating the whole thing as a job queue. It's most likely that by the time you get back a "job status code" lambda would already be done with the request. It seems to me API Gateway is really not what you need. SNS would be much more useful in a similar manner. You'd publish json messages, just like you've defined, to a certain topic1. This'd trigger subscriber lambda function. When the function is done and save the data to s3 it'd publish the result to another topic2. You may subsribe to topic2 beforehand to receive those messages. Cloudsearch with S3 would be just fine but I'm not sure if that's the best way to go for your use case. How exactly you want to search these files, by name, content, date, ect.? I expect this to take 10 days at most once the details are cleared.
$1,000 USD in 10 days
5.0 (14 reviews)
5.3
5.3
User Avatar
Hi I am certified aws solution architect with 3+ years of industry experience. I have been working on deploying highly scalable, available, fault-tolerant and optimized solutions using aws clouds. I actually found your project interesting one and want to work on this with you. Let us further discuss the project. I have expertise in following areas. 1. Microservice application development using aws lambda, api gateway, SAM, dynamodb etc. 2. Microservice deployment using Jenkin and Docker, Kuberneetes in aws ECS. 3. EC2(Autoscaling, load balancing, bootstrape automation) 4. RDS, SNS, SQS, SWF, Data analytics stack etc. 5. Shell, python, java, cron jobs. I can do this job for you according to your expectations. Looking forward to hearing from you. Best Regards
$500 USD in 10 days
5.0 (8 reviews)
4.1
4.1
User Avatar
Hi, I went through the requirement and what I feel is you need an async api which has to call multiple rest api's and store the result in some persistent store like dynamo db or S3. Couple of things which will help you to design your solution. 1) Use lambda (It supports execution upto 15mins now and multi threading is supported) 2) Use Dynamo DB instead of s3 for storing the result. It would be much convenient to do some basic CRUD operations in case you want on the reuslts 3) Use aws cloud formation to automate the entire deployment so that future changes can deployed easily. 4) Serverless application Model is awesome and simple. You must try for this task. Even if you are not interested in my proposal please give the above suggestions to the freelancer who is going to work on this. From my experience over 5 years in aws these things are definitely needed for your system in the long run,.
$497 USD in 10 days
4.8 (8 reviews)
3.8
3.8
User Avatar
******************************************************************************************************************************************************************************************************************************************************************************
$750 USD in 20 days
0.0 (0 reviews)
0.0
0.0
User Avatar
Hello there, We are a team of AWS experts with more than 12 years of rich industry experience .We provide DevOps as a service to wide variety of companies ranging from Small Scale to large organisations. Following are our DevOps services along with the plethora of DevOps tools we use : Container Services: Docker,Kubernetes,EcS,rkt Infrastructure provisioning:CloudFormation,Terraform Configuration Management:Ansible,Chef,PuppetLabs,Saltstack CI/CD:Concourse,AWS code pipeline,go,Jenkins Performance gathering:Collectd,Telegraf Metrics Aggregation and analytics:influxDb,graphana,Graphite Build system:Ant,Gradle,Grunt,Ivy,Maven,sbt Environment Building:VirtualBox Alert Management:Cloudwatch,Twilio,Nagios We can assure the best DevOps solutions at the best possible rates and minimal time frames. Thanks for reading the proposal. Regards Chhavi
$555 USD in 10 days
0.0 (0 reviews)
0.0
0.0
User Avatar
I'm working with AWS for 5 years, and currently studying to get my AWS Solutions Architect certification. I'm ready to solution your project and deliver a working cloud application as outlined in the provided doc. I do have some questions, when is a good time to speak by phone? If you award me this project I'd provide an architecture diagram and overview how I'm planning to complete the project. Once you OK it, I'll deliver you the POC, and last but not least the production ready solution.
$988 USD in 10 days
0.0 (0 reviews)
0.0
0.0

About the client

Flag of UNITED STATES
Waxhaw, United States
0.0
0
Member since Nov 4, 2018

Client Verification

Thanks! We’ve emailed you a link to claim your free credit.
Something went wrong while sending your email. Please try again.
Registered Users Total Jobs Posted
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Loading preview
Permission granted for Geolocation.
Your login session has expired and you have been logged out. Please log in again.