Exporting large data to CSV using mysql outfile in multi-host Rails env.

In one of our current project we had a requirement where we needed to allow the site admin to download raw DB data as CSV file for some offline analysis. So, according to the requirement, we allowed the admin to select a database table and a date range to download the generated CSV data.

Our initial approach was to send the CSV file asynchronously using delayed job. It worked like this:
Admin submits the request, system queues it and sends admin an email with the attached CSV file. The reason we did this asynchronously was to avoid timeout during CSV generation.

While testing & benchmarking this approach, we ecountered two issues when we had a large data to export.

  • Too much memory was used by the ruby process when generating the CSV
  • CSV size could be larger than the supported attachment size by the email service provider, hence could result in an undelivered email

Read more


Generate paperclip missing styles on the fly

Like most developers, who use paperclip gem with their RubyOnRails applications to support picture uploading and generation of distinct styles/thumbnails of pictures, for displaying them in various views, we also do the same in a lot of our apps.

After running one such web app in production for 2 years, the app needed to be redesigned. This required us to generate couple more styles/sizes for each uploaded image.

Paperclip provides a method 'reprocess!(:style)' to generate all/one style(s).

Now, when we planned to move this new version of the app to production, we had to generate those missing styles. There were a few thousand thumbnails to be generated, and our estimated time to generate these missing styles was around half an hour. Since we could not have the site available to users without the graphics, we had to put the site in maintenance mode until the new styles got generated. We also realized that as more images get uploaded and used in the system, this downtime if we ever needed to resize graphics again would go up. We could also not do this conversion without taking the site down as more images may get uploaded while we might be resizing using a script.
Read more


Full-Calendar Rails Engine

A couple of years ago we integrated jQuery FullCalendar plugin with a rails backend. The plugin became reasonably popular and was being used in many Rails apps.

Recently we upgraded it to include an efficient full-sized, drag & drop calendar functionality within a Rails Engine and release it as a gem. With Rails 4 deprecating Rails::Plugin(commit) this would come really handy.

It easily blends within a rails application. Provides a great UI experience with the use of AJAX to fetch events on-the-fly for each month and hooks for user-triggered events (like clicking or dragging an event) for an easy event editing functionality. The view can be easily customized using the configurations provided.

A demo version of this engine implementation can be viewed at http://vinsol.com/fullcalendar-demo

Work in progress : Optimizing Recurring Event functionality.

Source code is hosted on github. Feel free to fork and send a pull request etc.


Guide to Caching in Rails using Memcache

Guide to Caching in Rails using Memcache 1

The post intends to cover the topics and tools that can help in implementing Memcache as caching store in Rails and debugging issues with it.

To help newbies grasp it from start, it also provides references to installing and validating the memcache install without Rails too.

"Memcached is an in-memory key-value store for small chunks of arbitrary data (strings, objects) from results of database calls, API calls, or page rendering"

Installing Memcache

  • Mac OS X Snow Leopard
    • brew install memcached
  • Ubuntu
    • sudo apt-get install memcached

More information about Installing Memcache can be found here

Starting Memcache

  • memcached -vv

-v option controls verbosity to STDOUT/STDERR. Multiple -v's increase verbosity. Memcache Settings can be inspected in detail using echo "stats settings" | nc localhost 11211. Get the thirst quenched more at http://linux.die.net/man/1/memcached or man memcached for all options.

Connecting to Memcache Server through Telnet

  • telnet localhost 11211 can be used to connect locally to run/inspect memcache data store
  • Its recommended to follow Using Memcache Commands to get a full hang of memcache commands
  • For the sake of completeness, here are three main commands (store/retrieve/remove key/values)
    • Write to Memcache (Value need to be entered in new line)
      • set abc 0 60 5 # where abc is the key, 60 is duration in seconds and 5 is length of data
      • hello # data to store in abc key
    • Read from Memcache
      • get abc
    • Delete from Memcache
      • delete abc

Implementing and Using in Rails

Installing Dalli (Memcache Client)

Configure Memcache as cache store in Rails

  • Set perform_caching and specify cache_store in the environment file (e.g. RAILS_ROOT/config/environments/production.rb)
    • config.action_controller.perform_caching = true
    • config.cache_store = :dalli_store, { :namespace => "my_project", :expires_in => 1.day, :socket_timeout => 3, :compress => true }
      • Detailed explanation of Dalli Client options can be found here

Getting Ready for the Fight/Debugging

  • IRB as always helps us get started:
    • require 'rubygems'
    • require 'dalli'
    • CACHE = Dalli::Client.new('127.0.0.1', { :namespace => "my_project", :expires_in => 3600, :socket_timeout => 3, :compress => true }) # Options are self-explanatory
    • CACHE.set('key', 'value', 180) # last option is expiry time in seconds
    • CACHE.get('key') 2
  • Memcache does not provide any command to list all keys present but at times seeing the keys and their expiry time can be a life saviour. Here is a Ruby Script to list all memcache keys

    Sample Output:

    id expires bytes cache_key
    1 2014-01-22 19:05:09 +0530 5 my_project:key
    ...

Using Memcache to store expensive calculations and avoid recalculations

  • We can store expensive calculations in memcache and retrieve it next time if expensive calculation expected to return the same result else recalculate and store again.
  • Rails provides helpers like Rails.cache.read to read cache, Rails.cache.write to write in cache and Rails.cache.fetch to return result if present in cache else write in cache and return the result
  • Code Snippet to pass a block to evaluate if key not present in cache, else return the value of key

    
    
  • Using data_cache helper created in last step in application_controller.rb

    
    

Clearing Cache

Auto-Expire after a specified time

  • Memcache accepts an expiration time 3 option with the key to write and only returns the stored result if key has not expired yet, else returns nothing.
    • Rails.cache.write(key, value, {expires_in: time})
    • Rails.cache.fetch(key, {expires_in: time}) { # Block having expensive calculations }

Faking Regex based Expiry

  • Unfortunately, Memcache does not allow listing of all keys so, apparently, there is no way to partially expire a subset of keys. Memcache only provides a delete method for the specified key restricting us to, beforehand, know the key to delete.
  • But, there is a nice trick to partially expire a subset of keys Faking Regex Based Cache keys in Rails 4.
    • To achieve it, we have to focus more on keyword subset and defining subset is the Nirvana to crack the problem.
    • Introduce a dynamic component in all the keys which we want to treat as subset
    • Have a way to control the change of dynamic component
    • When dynamic component changes, a new key is created and stored in Memcache which would be referred for data from now.
    • The old key still remains in memcache and has still not expired but it stores the stale data and fresh data is hold by new key which we are referring in all of our interactions.
    • Memcache would eventually kick the old key with stale data out (uses LRU algorithm to allocate space for new data)
    • In the snippet below, we have added dynamic component last_cache_refreshed_at in Category which can be selectively changed to expire cache of particular categories instead of all

      
      

Writing Test Cases

Gotchas

  • Using touch command to update expiry cache time (Fixed in Memcache 1.4.14)
  • Check Memcache Version using telnet client
    • Execute stats
    • Look for STAT version 1.4.13

  1. Caching Expensive Api Requests Results

  2. If Object is saved in Memcache, Cache.get('key') might raise

    Dalli::UnmarshalError: Unable to unmarshal value: undefined class/module <model_name>

    Use require_dependency "app/models/.rb"

  3. Expiry Time of 0 makes the key stays for indefinite time only limited by Memcache clearing it out in case of cleaning up

  4. Scoped Expiration AND Reload Namespace using a Proc


'Launching Soon' – Helping Startups Build Online Identity Even before the Actual Launch

The process of brand building begins even before the actual website is launched. To have an eye-catching temporary ‘Launching Soon’ page can do a world of good for your online brand identity. It is the first step that lets you create initial buzz about your business and, therefore, it becomes imperative to have a dedicated ‘Launching Soon’ or ‘Coming Soon’ page in place. Needless to say that an aesthetic landing page embosses a long lasting and distinctive impression in the visitors’ psyche, ensuring a successful first step in climbing the long ladder.

Quick Introduction about ‘Launching Soon’

'Launching Soon' is a simple rails plugin that helps rails projects to manage a dedicated launching soon page before the actual launch date. The plugin also collects email from potential customers and can be integrated through an API to popular email marketing service providers like MailChimp & CampaignMonitor (to store customers email contacts). It lets you have a launch timer in place, and you can even have your company logo, about us section and much more. In a nutshell, all primary branding elements are present on your landing page (even before the launch of the official website). Exciting, isn’t it!

To hack over the excitement, ‘Launching Soon’ is an open source rails plugin which can be used by anyone & everyone to set up a fully customized and dedicated landing page in just a few minutes.

Why bother for a dedicated launch soon page?

Gone are the days when terrible yet oh so nostalgic ‘Under Construction’ animated GIF was appreciated by visitors. In its place, the use of more effective, functional and boiled down single ‘Coming Soon’ page is a cogent trend, capable of instantly satisfying the primary goals of the website owners.

For obvious reasons, there are other benefits of exerting efforts to have a great ‘Coming Soon’ page:

1.   Jumpstart your web presence – An effective landing page is the best way to ease the transition from going from naught to a full matured website launch.
2.    Engage with the visitors – Think from where you can benefit the most, a garish graphic which says coming soon or one page site that interacts with the visitors.
3.    Capture user data – This is perhaps one of the most important aspects of having a landing page. It lets you gather a database of genuinely interested users to whom you can send a newsletter notifying the release of your website. Big enough to help your site gain momentum.
4.    Search engine optimization – Getting a domain name booked is not enough. It is essential and imperative to have something substantial online immediately rather than wait for the final product.

With Launching Soon, it is possible to encompass all the above mentioned aspects without any rigorous development and design hassles. With a simple rails plug and play you would be up and running within minutes with your landing page, buzzing about your business and building an online identity. So, if you need a launching soon page then this is it…'Launching Soon' can be a near perfect solution.

Get the app code at:

https://github.com/vinsol/Launching-Soon

Note - Currently ‘Launching Soon’ works  with only rails 2.x. We are working on its compatibility with rails 3.x and would keep you posted with the updates.