• Linux
• Rsyslog
• Queues

Linux/Rsyslog/Queues

Message Flow

• See: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/s1-working_with_queues_in_rsyslog

• Flow:
  1. Msg received

  2. Pre-Processor -> Main message queue

  3. Rule-processor deque, run parsing and filtering engine, according to /etc/ryslog.conf
  4. Each action has its own action queue, that receives messaged for action processors to work on.
     • Several actions can run simultaneously on one msg.
     • action queue can by by passed, direct queues.
     • If action fails, action queue gets notified, and after some time will be attempted again by action.

  5. Recommended to use dedicated action queues for outputs to network/db to allow these to q off main q, preventing blocked main queue.

• Queue Types - Where type = MainMsg or action  $<object>QueueType <type> 

  1. direct - Default for action queue, perfect for writing to local file.  $ActionQueueType Direct 

  2. linkedlist - in-memory, recommended
  3. fixedarray - in-memory, 10k

  4. disk - Slow, reliable, 10Mb default, can set size and name-prefix  $objectQueueMaxFileSize size 

• Combine in-memory and disk with disk-assisted in-memory queue
  • Add settings to normal in-memory queue

    • $objectQueueFileName 

    • Set Watermark - number=# msg in queue, remember to replace <object> with MainMsg or Action

      •  $objectQueueHighWatermark number  ; $objectQueueLowWatermark number 

• MainMsg Queue

  • Default limit 10k
• Rsyslog log buffering

  • Location set by variable: $?WorkDirectory /var/lib/rsyslog

  • For forwarding to server e.g. example.com, with in-memory queue, and disk-assisted

    $ActionQueueType LinkedList
    $ActionQueueFileName example_fwd
    $ActionResumeRetryCount -1
    $ActionQueueSaveOnShutdown on
    *.*       @@example.com:6514

  • Or new style config prefered

    *.* action(type="omfwd"
          queue.type="linkedlist"
          queue.filename="example_fwd"
          action.resumeRetryCount="-1"
          queue.saveOnShutdown="on"
          target="example.com" port="6514" protocol="tcp"
         )

    • Msg's kept in memory, and when full disk queue /var/lib/ryslog/example_fd<number> used.