Find Jobs
Hire Freelancers

Logstash SQS Output Config

$30-250 USD

Closed
Posted over 8 years ago

$30-250 USD

Paid on delivery
I have a logstash based setup with files as inputs & SQS as [login to view URL], I have a JMS style listener on the Http service side which listens to any logs in the SQS & processes the same. The output SQS config is as shown below: output { sqs { queue => "test-queue" access_key_id => "ABCDE"  secret_access_key => "gfksfgsvcsakgcak" region => "us-east-1" } } However the issue arises as Logstash send @timestamp, @version fields to the SQS queue & the custom http service breaks as a result (specifically because of @timestamp field). Also, some other required fields are missing from SQS output configuration. After putting some thought, the way to handle this would be to modify the Logstash output SQS config to send messages in the below format (according to Http RFC 2616). A message would consist of header and a body just like an http message. Application:Foo \r\n Tenant:HelloWorld \r\n Table:Test \r\n Authorization:Basic xxxx \r\n \r\n {{"batch": { "docs": [..]}}} So, as you can see \r\n is used as a line break in the protocol specification for all header elements. Each header element is like a fieldname:fieldValue element. Additional \r\n in the begging of a blank line signified that what follows is the payload-body Please let me know for any additional questions. Desired Skills Amazon Web Services, Ruby, logstash, sqs
Project ID: 8532249

About the project

Remote project
Active 9 yrs ago

Looking to make some money?

Benefits of bidding on Freelancer

Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs

About the client

Flag of INDIA
Malerkotla, India
4.6
17
Payment method verified
Member since Dec 6, 2014

Client Verification

Thanks! We’ve emailed you a link to claim your free credit.
Something went wrong while sending your email. Please try again.
Registered Users Total Jobs Posted
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Loading preview
Permission granted for Geolocation.
Your login session has expired and you have been logged out. Please log in again.