014 RR Queues and Background Processing


    Definition of Queuing and Background Processes:

    • Queuing is about messaging. Typically first in first out (FIFO)
    • Background Processes are processes that pull messages off the queue and do things later.

    Systems that people have used

    Poor man’s queue or database backed queue What do you look for in your queue technology?

    • How it takes failure
    • Introspection
    • It depends on what you’re doing
      • Regular tasks vs. Immediate tasks
      • One worker vs several workers on several servers

    Dave’s beanstalk utilities Beanstalk RailsCast Hybrid approaches

    • Google’s Geocoding API – Rate-limiting with cron
    • Amazon’s FPS – polling amazon to get updates on payments

    Best Practices

    • Structure jobs so they are simple input/output
    • Isolate your jobs as much as possible from the database
    • Decouple your application from your queue – Message Passing
    • Distributed == Not Dependent – If the job dies and you are broken, you’re designed wrong
    • Idempotence on the job
    • Messaging Queue == State Machine
    • Logging/Emailing when something is unprocessed for too long

    What do you send to the background?

    • Payment Processing
    • Emails
    • PDF Generation
    • Billing reconciliation
    • Geocoding
    • External Web Services
    • Collate events into an aggregate event
    • Farming out multiple jobs to multiple queues
    • Separate unusual resources onto other servers