Exporting large data to CSV using mysql outfile in multi-host Rails env.

In one of our current project we had a requirement where we needed to allow the site admin to download raw DB data as CSV file for some offline analysis. So, according to the requirement, we allowed the admin to select a database table and a date range to download the generated CSV data. Our initial approach was to send the CSV file asynchronously using delayed job. It worked like this: Admin submits the request, system queues it and sends admin an email with the attached CSV file. The reason we did this asynchronously was to avoid timeout during CSV generation. While testing & benchmarking this approach,…