The Comms Room

« first day (1845 days earlier) ← previous day next day → last day (3136 days later) »

MrPHP

3:03 AM

hii

3 hours later…

Dawny33

5:35 AM

Hello, I am a newbie here. I wanted to know whether this question is on-topic here or not.

"How do I build a pipeline for updating data from Logstash to Redshift real-time"?

I have been asked to post the question here, after a discussion on the SE meta
(http://meta.stackexchange.com/questions/267043/where-would-this-question-about-data-pipelines-belong)

Journeyman Geek

@Dawny33: Generally a great question talks about what you've tried and what's the specific issue you have. Least as it is, seems kinda broad

"You seem to have come in with a preconceived idea of the solution - why not ask about the actual problem and let the experts inform you of the best solution?" Seems an awesome bit of advice too

Dawny33

I have tried the ElasticSearch output plugin for S3. But, it does it in batches.

Thank you. So, can you please let me know how do I frame the question? All my front-end analytics data is on Redshift, and I want to store my logs(from the ELK stack) on Redshift, for data science purpose.

Journeyman Geek

Taking into account the site rules, If I were asking that question, I'd probably frame it in terms of what I'm trying to do. As always give enough detail, talk about both ends of the issue, and what you're trying to do I guess

(so totally read that /help page)

Iain

@Dawny33 That looks like a product recommendation - it'll just get closed off topic

Dawny33

@Iain I have seen your comment.

But, the question looks like a product recommendation. However, it is not so. I am asking for a solution for storing the logs data realtime in Redshift.

But, seems like I have to look into the help page, before asking the question. Thank you @JourneymanGeek

Journeyman Geek

5:42 AM

Or that, I suppose

Dawny33

5:59 AM

@JourneymanGeek

Posted the question. Can you please let me know, whether it is properly framed?

Journeyman Geek

@Dawny33: What do the source and endpoint run? What OSes?

8 hours later…