I have a logstash based setup with files as inputs & SQS as [login to view URL], I have a JMS style listener on the Http service side which listens to any logs in the SQS & processes the same. The output SQS config is as shown below:
output {
sqs {
queue => "test-queue"
access_key_id => "ABCDE"
secret_access_key => "gfksfgsvcsakgcak"
region => "us-east-1"
}
}
However the issue arises as Logstash send @timestamp, @version fields to the SQS queue & the custom http service breaks as a result (specifically because of @timestamp field). Also, some other required fields are missing from SQS output configuration.
After putting some thought, the way to handle this would be to modify the Logstash output SQS config to send messages in the below format (according to Http RFC 2616). A message would consist of header and a body just like an http message.
Application:Foo \r\n
Tenant:HelloWorld \r\n
Table:Test \r\n
Authorization:Basic xxxx \r\n
\r\n
{{"batch": { "docs": [..]}}}
So, as you can see \r\n is used as a line break in the protocol specification for all header elements. Each header element is like a fieldname:fieldValue element. Additional \r\n in the begging of a blank line signified that what follows is the payload-body
Please let me know for any additional questions.
Desired Skills
Amazon Web Services, Ruby, logstash, sqs