Mar 19 23:45:01 cheezum /USR/SBIN/CRON[12667]: (root) CMD ([ -x /usr/lib/sysstat/sa1 ] && { [ -r "$DEFAULT" ] && . "$DEFAULT" ; [ "$ENABLED" = "true" ] && exec /usr/lib/sysstat/sa1 $SA1_OPTIONS 1 1 ; })
Mar 19 23:55:01 cheezum /USR/SBIN/CRON[12669]: (root) CMD ([ -x /usr/lib/sysstat/sa1 ] && { [ -r "$DEFAULT" ] && . "$DEFAULT" ; [ "$ENABLED" = "true" ] && exec /usr/lib/sysstat/sa1 $SA1_OPTIONS 1 1 ; })
Mar 19 23:59:01 cheezum /USR/SBIN/CRON[12671]: (root) CMD ([ -x /usr/lib/sysstat/sa1 ] && { [ -r "$DEFAULT" ] && . "$DEFAULT" ; [ "$ENABLED" = "true" ] && exec /usr/lib/sysstat/sa1 $SA1_OPTIONS 60 2 ; })
Mar 20 00:00:52 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=60003 jiffies)
Mar 20 00:00:52 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 00:00:52 cheezum kernel: Call Trace:
Mar 20 00:00:52 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 00:00:52 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 00:00:52 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 00:00:52 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 00:00:52 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:00:52 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:00:52 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:00:52 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:00:52 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:00:52 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:00:52 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:00:52 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:00:52 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:00:52 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:00:52 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:00:52 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:00:52 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:00:52 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:00:52 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:00:52 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:00:52 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 00:00:52 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 00:00:52 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 00:00:52 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=60000 jiffies)
Mar 20 00:00:52 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 00:00:52 cheezum kernel: Call Trace:
Mar 20 00:00:52 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 00:00:52 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 00:00:52 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:00:52 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:00:52 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:00:52 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:00:52 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:00:52 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:00:52 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:00:52 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:00:52 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:00:52 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:00:52 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:00:52 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:00:52 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:00:52 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:00:52 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:00:52 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:00:52 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 00:00:52 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 00:00:52 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:00:52 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 00:00:52 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 00:00:52 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 00:00:52 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 00:00:52 cheezum kernel:  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 00:00:52 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 00:00:52 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 00:00:52 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 00:00:52 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 00:00:52 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 00:00:52 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 00:00:52 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 00:00:52 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 00:00:52 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 00:00:52 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 00:03:52 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=240035 jiffies)
Mar 20 00:03:52 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 00:03:52 cheezum kernel: Call Trace:
Mar 20 00:03:52 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 00:03:52 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 00:03:52 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 00:03:52 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 00:03:52 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:03:52 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:03:52 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:03:52 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:03:52 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:03:52 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:03:52 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:03:52 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:03:52 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:03:52 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:03:52 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:03:52 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:03:52 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:03:52 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:03:52 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:03:52 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:03:52 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 00:03:52 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 00:03:52 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 00:03:52 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=240030 jiffies)
Mar 20 00:03:52 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 00:03:52 cheezum kernel: Call Trace:
Mar 20 00:03:52 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 00:03:52 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 00:03:52 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:03:52 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:03:52 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:03:52 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:03:52 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:03:52 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:03:52 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:03:52 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:03:52 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:03:52 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:03:52 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:03:52 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:03:52 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:03:52 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:03:52 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:03:52 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:03:52 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 00:03:52 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 00:03:52 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:03:52 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 00:03:52 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 00:03:52 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 00:03:52 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 00:03:52 cheezum kernel:  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 00:03:52 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 00:03:52 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 00:03:52 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 00:03:52 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 00:03:52 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 00:03:52 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 00:03:52 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 00:03:52 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 00:03:52 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 00:03:52 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 00:05:01 cheezum /USR/SBIN/CRON[12679]: (root) CMD ([ -x /usr/lib/sysstat/sa1 ] && { [ -r "$DEFAULT" ] && . "$DEFAULT" ; [ "$ENABLED" = "true" ] && exec /usr/lib/sysstat/sa1 $SA1_OPTIONS 1 1 ; })
Mar 20 00:06:52 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=420067 jiffies)
Mar 20 00:06:52 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 00:06:52 cheezum kernel: Call Trace:
Mar 20 00:06:52 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 00:06:52 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 00:06:52 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 00:06:52 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 00:06:52 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:06:52 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:06:52 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:06:52 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:06:52 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:06:52 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:06:52 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:06:52 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:06:52 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:06:52 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:06:52 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:06:52 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:06:52 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:06:52 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:06:52 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:06:52 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:06:52 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 00:06:52 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 00:06:52 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 00:06:52 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=420060 jiffies)
Mar 20 00:06:52 cheezum kernel: INFO: rcu_sched detected stall on CPU 0 (t=420060 jiffies)
Mar 20 00:06:52 cheezum kernel: Pid: 0, comm: swapper/0 Not tainted 3.2.1-linode40 #1
Mar 20 00:06:52 cheezum kernel: Call Trace:
Mar 20 00:06:52 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 00:06:52 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 00:06:52 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:06:52 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:06:52 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 00:06:52 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:06:52 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:06:52 cheezum kernel: Call Trace:
Mar 20 00:06:52 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:06:52 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:06:52 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 00:06:52 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 00:06:52 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:06:52 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:06:52 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:06:52 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:06:52 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:06:52 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:06:52 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:06:52 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:06:52 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:06:52 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:06:52 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:06:52 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:06:52 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:06:52 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 00:06:52 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:06:52 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 00:06:52 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:06:52 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 00:06:52 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:06:52 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 00:06:52 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 00:06:52 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 00:06:52 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:06:52 cheezum kernel:  [<c05de8ba>] ? rt_garbage_collect+0x12a/0x400
Mar 20 00:06:52 cheezum kernel:  [<c0564add>] ? dst_alloc+0xfd/0x140
Mar 20 00:06:52 cheezum kernel:  [<c05dfc97>] ? rt_dst_alloc+0x37/0x40
Mar 20 00:06:52 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:06:52 cheezum kernel:  [<c05e16cf>] ? ip_route_input_slow+0x28f/0x560
Mar 20 00:06:52 cheezum kernel:  [<c05e0010>] ? __mkroute_output+0x10/0x2e0
Mar 20 00:06:52 cheezum kernel:  [<c059a2e4>] ? nf_hook_slow+0x64/0x130
Mar 20 00:06:52 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:06:52 cheezum kernel:  [<c05e3213>] ? ip_rcv_finish+0x243/0x380
Mar 20 00:06:52 cheezum kernel:  [<c055bad8>] ? __netif_receive_skb+0x2e8/0x350
Mar 20 00:06:52 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:06:52 cheezum kernel:  [<c055d3f7>] ? netif_receive_skb+0x67/0x70
Mar 20 00:06:52 cheezum kernel:  [<c04fcdb1>] ? handle_incoming_queue+0x1a1/0x270
Mar 20 00:06:52 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:06:52 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:06:52 cheezum kernel:  [<c04fd0a4>] ? xennet_poll+0x224/0x570
Mar 20 00:06:52 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:06:52 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:06:52 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:06:52 cheezum kernel:  [<c055db8a>] ? net_rx_action+0xea/0x190
Mar 20 00:06:52 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:06:52 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 00:06:52 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 00:06:52 cheezum kernel:  [<c013ec5c>] ? __do_softirq+0x7c/0x110
Mar 20 00:06:52 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:06:52 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 00:06:52 cheezum kernel:  [<c013ebe0>] ? irq_enter+0x60/0x60
Mar 20 00:06:52 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 00:06:52 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 00:06:52 cheezum kernel:  <IRQ>  [<c013ea3e>] ? irq_exit+0x6e/0xa0
Mar 20 00:06:52 cheezum kernel:  [<c048d9dd>] ? xen_evtchn_do_upcall+0x1d/0x30
Mar 20 00:06:52 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:06:52 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:06:52 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 00:06:52 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 00:06:52 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 00:06:52 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 00:06:52 cheezum kernel:  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 00:06:52 cheezum kernel:  [<c08837d1>] ? start_kernel+0x2dd/0x2e2
Mar 20 00:06:52 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 00:06:52 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 00:06:52 cheezum kernel:  [<c0883332>] ? kernel_init+0x141/0x141
Mar 20 00:06:52 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 00:06:52 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 00:06:52 cheezum kernel:  [<c088672b>] ? xen_start_kernel+0x4f7/0x4ff
Mar 20 00:06:52 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 00:06:52 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 00:06:52 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 00:06:52 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 00:06:52 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 00:06:52 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 00:09:52 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=600099 jiffies)
Mar 20 00:09:52 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 00:09:52 cheezum kernel: Call Trace:
Mar 20 00:09:52 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 00:09:52 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 00:09:52 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 00:09:52 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 00:09:52 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:09:52 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:09:52 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:09:52 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:09:52 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:09:52 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:09:52 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:09:52 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:09:52 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:09:52 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:09:52 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:09:52 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:09:52 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:09:52 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:09:52 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:09:52 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:09:52 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 00:09:52 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 00:09:52 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 00:09:52 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=600090 jiffies)
Mar 20 00:09:52 cheezum kernel: INFO: rcu_sched detected stall on CPU 0 (t=600090 jiffies)
Mar 20 00:09:52 cheezum kernel: Pid: 0, comm: swapper/0 Not tainted 3.2.1-linode40 #1
Mar 20 00:09:52 cheezum kernel: Call Trace:
Mar 20 00:09:52 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 00:09:52 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 00:09:52 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 00:09:52 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:09:52 cheezum kernel: Call Trace:
Mar 20 00:09:52 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:09:52 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:09:52 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:09:52 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:09:52 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 00:09:52 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 00:09:52 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:09:52 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:09:52 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:09:52 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:09:52 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:09:52 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:09:52 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:09:52 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:09:52 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:09:52 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:09:52 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:09:52 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:09:52 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:09:52 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:09:52 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:09:52 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:09:52 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:09:52 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:09:52 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:09:52 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 00:09:52 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 00:09:52 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:09:52 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 00:09:52 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:09:52 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 00:09:52 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 00:09:52 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 00:09:52 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:09:52 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:09:52 cheezum kernel:  [<c05de8ba>] ? rt_garbage_collect+0x12a/0x400
Mar 20 00:09:52 cheezum kernel:  [<c0564add>] ? dst_alloc+0xfd/0x140
Mar 20 00:09:52 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:09:52 cheezum kernel:  [<c05dfc97>] ? rt_dst_alloc+0x37/0x40
Mar 20 00:09:52 cheezum kernel:  [<c05e16cf>] ? ip_route_input_slow+0x28f/0x560
Mar 20 00:09:52 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:09:52 cheezum kernel:  [<c05e0010>] ? __mkroute_output+0x10/0x2e0
Mar 20 00:09:52 cheezum kernel:  [<c059a2e4>] ? nf_hook_slow+0x64/0x130
Mar 20 00:09:52 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:09:52 cheezum kernel:  [<c05e3213>] ? ip_rcv_finish+0x243/0x380
Mar 20 00:09:52 cheezum kernel:  [<c055bad8>] ? __netif_receive_skb+0x2e8/0x350
Mar 20 00:09:52 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:09:52 cheezum kernel:  [<c055d3f7>] ? netif_receive_skb+0x67/0x70
Mar 20 00:09:52 cheezum kernel:  [<c04fcdb1>] ? handle_incoming_queue+0x1a1/0x270
Mar 20 00:09:52 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:09:52 cheezum kernel:  [<c04fd0a4>] ? xennet_poll+0x224/0x570
Mar 20 00:09:52 cheezum kernel:  [<c055db8a>] ? net_rx_action+0xea/0x190
Mar 20 00:09:52 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 00:09:52 cheezum kernel:  [<c013ec5c>] ? __do_softirq+0x7c/0x110
Mar 20 00:09:52 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 00:09:52 cheezum kernel:  [<c013ebe0>] ? irq_enter+0x60/0x60
Mar 20 00:09:52 cheezum kernel:  <IRQ>  [<c013ea3e>] ? irq_exit+0x6e/0xa0
Mar 20 00:09:52 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:09:52 cheezum kernel:  [<c048d9dd>] ? xen_evtchn_do_upcall+0x1d/0x30
Mar 20 00:09:52 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:09:52 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 00:09:52 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:09:52 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 00:09:52 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 00:09:52 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 00:09:52 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 00:09:52 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 00:09:52 cheezum kernel:  [<c08837d1>] ? start_kernel+0x2dd/0x2e2
Mar 20 00:09:52 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 00:09:52 cheezum kernel:  [<c0883332>] ? kernel_init+0x141/0x141
Mar 20 00:09:52 cheezum kernel:  [<c088672b>] ? xen_start_kernel+0x4f7/0x4ff
Mar 20 00:09:52 cheezum kernel:  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 00:09:52 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 00:09:52 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 00:09:52 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 00:09:52 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 00:09:52 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 00:09:52 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 00:09:52 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 00:09:52 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 00:09:52 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 00:09:52 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 00:12:52 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=780131 jiffies)
Mar 20 00:12:52 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 00:12:52 cheezum kernel: Call Trace:
Mar 20 00:12:52 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 00:12:52 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 00:12:52 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 00:12:52 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 00:12:52 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:12:52 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:12:52 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:12:52 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:12:52 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:12:52 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:12:52 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:12:52 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:12:52 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:12:52 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:12:52 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:12:52 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:12:52 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:12:52 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:12:52 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:12:52 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:12:52 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 00:12:52 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 00:12:52 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 00:12:52 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=780120 jiffies)
Mar 20 00:12:52 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 00:12:52 cheezum kernel: Call Trace:
Mar 20 00:12:52 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 00:12:52 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 00:12:52 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:12:52 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:12:52 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:12:52 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:12:52 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:12:52 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:12:52 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:12:52 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:12:52 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:12:52 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:12:52 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:12:52 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:12:52 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:12:52 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:12:52 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:12:52 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:12:52 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 00:12:52 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 00:12:52 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:12:52 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 00:12:52 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 00:12:52 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 00:12:52 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 00:12:52 cheezum kernel:  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 00:12:52 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 00:12:52 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 00:12:52 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 00:12:52 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 00:12:52 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 00:12:52 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 00:12:52 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 00:12:52 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 00:12:52 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 00:12:52 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 00:15:01 cheezum /USR/SBIN/CRON[12692]: (root) CMD ([ -x /usr/lib/sysstat/sa1 ] && { [ -r "$DEFAULT" ] && . "$DEFAULT" ; [ "$ENABLED" = "true" ] && exec /usr/lib/sysstat/sa1 $SA1_OPTIONS 1 1 ; })
Mar 20 00:15:52 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=960163 jiffies)
Mar 20 00:15:52 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 00:15:52 cheezum kernel: Call Trace:
Mar 20 00:15:52 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 00:15:52 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 00:15:52 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 00:15:52 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 00:15:52 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:15:52 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:15:52 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:15:52 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:15:52 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:15:52 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:15:52 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:15:52 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:15:52 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:15:52 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:15:52 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:15:52 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:15:52 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:15:52 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:15:52 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:15:52 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:15:52 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 00:15:52 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 00:15:52 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 00:15:52 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=960150 jiffies)
Mar 20 00:15:52 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 00:15:52 cheezum kernel: Call Trace:
Mar 20 00:15:52 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 00:15:52 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 00:15:52 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:15:52 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:15:52 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:15:52 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:15:52 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:15:52 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:15:52 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:15:52 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:15:52 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:15:52 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:15:52 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:15:52 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:15:52 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:15:52 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:15:52 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:15:52 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:15:52 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 00:15:52 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 00:15:52 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:15:52 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 00:15:52 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 00:15:52 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 00:15:52 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 00:15:52 cheezum kernel:  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 00:15:52 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 00:15:52 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 00:15:52 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 00:15:52 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 00:15:52 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 00:15:52 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 00:15:52 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 00:15:52 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 00:15:52 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 00:15:52 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 00:17:01 cheezum /USR/SBIN/CRON[12699]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Mar 20 00:18:52 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=1140195 jiffies)
Mar 20 00:18:52 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 00:18:52 cheezum kernel: Call Trace:
Mar 20 00:18:52 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 00:18:52 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 00:18:52 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 00:18:52 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 00:18:52 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:18:52 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:18:52 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:18:52 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:18:52 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:18:52 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:18:52 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:18:52 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:18:52 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:18:52 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:18:52 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:18:52 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:18:52 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:18:52 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:18:52 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:18:52 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:18:52 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 00:18:52 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 00:18:52 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 00:18:52 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=1140180 jiffies)
Mar 20 00:18:52 cheezum kernel: INFO: rcu_sched detected stall on CPU 0 (t=1140180 jiffies)
Mar 20 00:18:52 cheezum kernel: Pid: 0, comm: swapper/0 Not tainted 3.2.1-linode40 #1
Mar 20 00:18:52 cheezum kernel: Call Trace:
Mar 20 00:18:52 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 00:18:52 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 00:18:52 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:18:52 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:18:52 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 00:18:52 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:18:52 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:18:52 cheezum kernel: Call Trace:
Mar 20 00:18:52 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:18:52 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:18:52 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 00:18:52 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:18:52 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:18:52 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:18:52 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 00:18:52 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:18:52 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:18:52 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:18:52 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:18:52 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:18:52 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:18:52 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:18:52 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:18:52 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:18:52 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:18:52 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 00:18:52 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:18:52 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 00:18:52 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:18:52 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:18:52 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 00:18:52 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:18:52 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:18:52 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:18:52 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 00:18:52 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:18:52 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:18:52 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:18:52 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 00:18:52 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:18:52 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 00:18:52 cheezum kernel:  [<c05de8ba>] ? rt_garbage_collect+0x12a/0x400
Mar 20 00:18:52 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:18:52 cheezum kernel:  [<c0564add>] ? dst_alloc+0xfd/0x140
Mar 20 00:18:52 cheezum kernel:  [<c05dfc97>] ? rt_dst_alloc+0x37/0x40
Mar 20 00:18:52 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:18:52 cheezum kernel:  [<c05e16cf>] ? ip_route_input_slow+0x28f/0x560
Mar 20 00:18:52 cheezum kernel:  [<c05e0010>] ? __mkroute_output+0x10/0x2e0
Mar 20 00:18:52 cheezum kernel:  [<c059a2e4>] ? nf_hook_slow+0x64/0x130
Mar 20 00:18:52 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:18:52 cheezum kernel:  [<c05e3213>] ? ip_rcv_finish+0x243/0x380
Mar 20 00:18:52 cheezum kernel:  [<c055bad8>] ? __netif_receive_skb+0x2e8/0x350
Mar 20 00:18:52 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:18:52 cheezum kernel:  [<c055d3f7>] ? netif_receive_skb+0x67/0x70
Mar 20 00:18:52 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 00:18:52 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 00:18:52 cheezum kernel:  [<c04fcdb1>] ? handle_incoming_queue+0x1a1/0x270
Mar 20 00:18:52 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:18:52 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 00:18:52 cheezum kernel:  [<c04fd0a4>] ? xennet_poll+0x224/0x570
Mar 20 00:18:52 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 00:18:52 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 00:18:52 cheezum kernel:  [<c055db8a>] ? net_rx_action+0xea/0x190
Mar 20 00:18:52 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 00:18:52 cheezum kernel:  [<c013ec5c>] ? __do_softirq+0x7c/0x110
Mar 20 00:18:52 cheezum kernel:  [<c013ebe0>] ? irq_enter+0x60/0x60
Mar 20 00:18:52 cheezum kernel:  <IRQ>  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 00:18:52 cheezum kernel:  [<c013ea3e>] ? irq_exit+0x6e/0xa0
Mar 20 00:18:52 cheezum kernel:  [<c048d9dd>] ? xen_evtchn_do_upcall+0x1d/0x30
Mar 20 00:18:52 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 00:18:52 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:18:52 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:18:52 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 00:18:52 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 00:18:52 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 00:18:52 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 00:18:52 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 00:18:52 cheezum kernel:  [<c08837d1>] ? start_kernel+0x2dd/0x2e2
Mar 20 00:18:52 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 00:18:52 cheezum kernel:  [<c0883332>] ? kernel_init+0x141/0x141
Mar 20 00:18:52 cheezum kernel:  [<c088672b>] ? xen_start_kernel+0x4f7/0x4ff
Mar 20 00:18:52 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 00:18:52 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 00:18:52 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 00:18:52 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 00:18:52 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 00:18:52 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 00:21:52 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=1320228 jiffies)
Mar 20 00:21:52 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 00:21:52 cheezum kernel: Call Trace:
Mar 20 00:21:52 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 00:21:52 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 00:21:52 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 00:21:52 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 00:21:52 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:21:52 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:21:52 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:21:52 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:21:52 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:21:52 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:21:52 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:21:52 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:21:52 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:21:52 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:21:52 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:21:52 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:21:52 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:21:52 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:21:52 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:21:52 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:21:52 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 00:21:52 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 00:21:52 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 00:21:52 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=1320210 jiffies)
Mar 20 00:21:52 cheezum kernel: INFO: rcu_sched detected stall on CPU 0 (t=1320210 jiffies)
Mar 20 00:21:52 cheezum kernel: Pid: 0, comm: swapper/0 Not tainted 3.2.1-linode40 #1
Mar 20 00:21:52 cheezum kernel: Call Trace:
Mar 20 00:21:52 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 00:21:52 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 00:21:52 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 00:21:52 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:21:52 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:21:52 cheezum kernel: Call Trace:
Mar 20 00:21:52 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:21:52 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:21:52 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:21:52 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 00:21:52 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:21:52 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 00:21:52 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:21:52 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:21:52 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:21:52 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:21:52 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:21:52 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:21:52 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:21:52 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:21:52 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:21:52 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:21:52 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:21:52 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:21:52 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:21:52 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:21:52 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:21:52 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:21:52 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:21:52 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:21:52 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:21:52 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:21:52 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:21:52 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:21:52 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:21:52 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:21:52 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:21:52 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:21:52 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 00:21:52 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 00:21:52 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 00:21:52 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 00:21:52 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:21:52 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 00:21:52 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:21:52 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 00:21:52 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 00:21:52 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 00:21:52 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 00:21:52 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 00:21:52 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 00:21:52 cheezum kernel:  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 00:21:52 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 00:21:52 cheezum kernel:  [<c05de8ba>] ? rt_garbage_collect+0x12a/0x400
Mar 20 00:21:52 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 00:21:52 cheezum kernel:  [<c0564add>] ? dst_alloc+0xfd/0x140
Mar 20 00:21:52 cheezum kernel:  [<c05dfc97>] ? rt_dst_alloc+0x37/0x40
Mar 20 00:21:52 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 00:21:52 cheezum kernel:  [<c05e16cf>] ? ip_route_input_slow+0x28f/0x560
Mar 20 00:21:52 cheezum kernel:  [<c05e0010>] ? __mkroute_output+0x10/0x2e0
Mar 20 00:21:52 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 00:21:52 cheezum kernel:  [<c059a2e4>] ? nf_hook_slow+0x64/0x130
Mar 20 00:21:52 cheezum kernel:  [<c05e3213>] ? ip_rcv_finish+0x243/0x380
Mar 20 00:21:52 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 00:21:52 cheezum kernel:  [<c055bad8>] ? __netif_receive_skb+0x2e8/0x350
Mar 20 00:21:52 cheezum kernel:  [<c055d3f7>] ? netif_receive_skb+0x67/0x70
Mar 20 00:21:52 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 00:21:52 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 00:21:52 cheezum kernel:  [<c04fcdb1>] ? handle_incoming_queue+0x1a1/0x270
Mar 20 00:21:52 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 00:21:52 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 00:21:52 cheezum kernel:  [<c04fd0a4>] ? xennet_poll+0x224/0x570
Mar 20 00:21:52 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 00:21:52 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 00:21:52 cheezum kernel:  [<c055db8a>] ? net_rx_action+0xea/0x190
Mar 20 00:21:52 cheezum kernel:  [<c013ec5c>] ? __do_softirq+0x7c/0x110
Mar 20 00:21:52 cheezum kernel:  [<c013ebe0>] ? irq_enter+0x60/0x60
Mar 20 00:21:52 cheezum kernel:  <IRQ>  [<c013ea3e>] ? irq_exit+0x6e/0xa0
Mar 20 00:21:52 cheezum kernel:  [<c048d9dd>] ? xen_evtchn_do_upcall+0x1d/0x30
Mar 20 00:21:52 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:21:52 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:21:52 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 00:21:52 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 00:21:52 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 00:21:52 cheezum kernel:  [<c08837d1>] ? start_kernel+0x2dd/0x2e2
Mar 20 00:21:52 cheezum kernel:  [<c0883332>] ? kernel_init+0x141/0x141
Mar 20 00:21:52 cheezum kernel:  [<c088672b>] ? xen_start_kernel+0x4f7/0x4ff
Mar 20 00:23:36 cheezum ntpd[3070]: i/o error on routing socket No buffer space available - disabling
Mar 20 00:24:52 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=1500260 jiffies)
Mar 20 00:24:52 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 00:24:52 cheezum kernel: Call Trace:
Mar 20 00:24:52 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 00:24:52 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 00:24:52 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 00:24:52 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 00:24:52 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:24:52 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:24:52 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:24:52 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:24:52 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:24:52 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:24:52 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:24:52 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:24:52 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:24:52 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:24:52 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:24:52 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:24:52 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:24:52 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:24:52 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:24:52 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:24:52 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 00:24:52 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 00:24:52 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 00:24:52 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=1500240 jiffies)
Mar 20 00:24:52 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 00:24:52 cheezum kernel: Call Trace:
Mar 20 00:24:52 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 00:24:52 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 00:24:52 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:24:52 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:24:52 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:24:52 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:24:52 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:24:52 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:24:52 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:24:52 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:24:52 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:24:52 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:24:52 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:24:52 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:24:52 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:24:52 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:24:52 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:24:52 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:24:52 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 00:24:52 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 00:24:52 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:24:52 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 00:24:52 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 00:24:52 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 00:24:52 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 00:24:52 cheezum kernel:  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 00:24:52 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 00:24:52 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 00:24:52 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 00:24:52 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 00:24:52 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 00:24:52 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 00:24:52 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 00:24:52 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 00:24:52 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 00:24:52 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 00:25:01 cheezum /USR/SBIN/CRON[12708]: (root) CMD ([ -x /usr/lib/sysstat/sa1 ] && { [ -r "$DEFAULT" ] && . "$DEFAULT" ; [ "$ENABLED" = "true" ] && exec /usr/lib/sysstat/sa1 $SA1_OPTIONS 1 1 ; })
Mar 20 00:27:52 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=1680293 jiffies)
Mar 20 00:27:52 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 00:27:52 cheezum kernel: Call Trace:
Mar 20 00:27:52 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 00:27:52 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 00:27:52 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 00:27:52 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 00:27:52 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:27:52 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:27:52 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:27:52 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:27:52 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:27:52 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:27:52 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:27:52 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:27:52 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:27:52 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:27:52 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:27:52 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:27:52 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:27:52 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:27:52 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:27:52 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:27:52 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 00:27:52 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 00:27:52 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 00:27:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=1680270 jiffies)
Mar 20 00:27:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 0 (t=1680270 jiffies)
Mar 20 00:27:53 cheezum kernel: Pid: 0, comm: swapper/0 Not tainted 3.2.1-linode40 #1
Mar 20 00:27:53 cheezum kernel: Call Trace:
Mar 20 00:27:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 00:27:53 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 00:27:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 00:27:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:27:53 cheezum kernel: Call Trace:
Mar 20 00:27:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 00:27:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:27:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 00:27:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:27:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:27:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:27:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:27:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:27:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:27:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:27:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:27:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:27:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:27:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:27:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:27:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:27:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:27:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:27:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:27:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:27:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:27:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:27:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:27:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:27:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:27:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:27:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:27:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:27:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 00:27:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 00:27:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:27:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:27:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:27:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:27:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 00:27:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:27:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:27:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 00:27:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 00:27:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 00:27:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 00:27:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:27:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 00:27:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 00:27:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 00:27:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 00:27:53 cheezum kernel:  [<c05de8ba>] ? rt_garbage_collect+0x12a/0x400
Mar 20 00:27:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 00:27:53 cheezum kernel:  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 00:27:53 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 00:27:53 cheezum kernel:  [<c0564add>] ? dst_alloc+0xfd/0x140
Mar 20 00:27:53 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 00:27:53 cheezum kernel:  [<c05dfc97>] ? rt_dst_alloc+0x37/0x40
Mar 20 00:27:53 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 00:27:53 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 00:27:53 cheezum kernel:  [<c05e16cf>] ? ip_route_input_slow+0x28f/0x560
Mar 20 00:27:53 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 00:27:53 cheezum kernel:  [<c05e0010>] ? __mkroute_output+0x10/0x2e0
Mar 20 00:27:53 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 00:27:53 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 00:27:53 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 00:27:53 cheezum kernel:  [<c059a2e4>] ? nf_hook_slow+0x64/0x130
Mar 20 00:27:53 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 00:27:53 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 00:27:53 cheezum kernel:  [<c05e3213>] ? ip_rcv_finish+0x243/0x380
Mar 20 00:27:53 cheezum kernel:  [<c055bad8>] ? __netif_receive_skb+0x2e8/0x350
Mar 20 00:27:53 cheezum kernel:  [<c055d3f7>] ? netif_receive_skb+0x67/0x70
Mar 20 00:27:53 cheezum kernel:  [<c04fcdb1>] ? handle_incoming_queue+0x1a1/0x270
Mar 20 00:27:53 cheezum kernel:  [<c04fd0a4>] ? xennet_poll+0x224/0x570
Mar 20 00:27:53 cheezum kernel:  [<c055db8a>] ? net_rx_action+0xea/0x190
Mar 20 00:27:53 cheezum kernel:  [<c013ec5c>] ? __do_softirq+0x7c/0x110
Mar 20 00:27:53 cheezum kernel:  [<c013ebe0>] ? irq_enter+0x60/0x60
Mar 20 00:27:53 cheezum kernel:  <IRQ>  [<c013ea3e>] ? irq_exit+0x6e/0xa0
Mar 20 00:27:53 cheezum kernel:  [<c048d9dd>] ? xen_evtchn_do_upcall+0x1d/0x30
Mar 20 00:27:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:27:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:27:53 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 00:27:53 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 00:27:53 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 00:27:53 cheezum kernel:  [<c08837d1>] ? start_kernel+0x2dd/0x2e2
Mar 20 00:27:53 cheezum kernel:  [<c0883332>] ? kernel_init+0x141/0x141
Mar 20 00:27:53 cheezum kernel:  [<c088672b>] ? xen_start_kernel+0x4f7/0x4ff
Mar 20 00:30:52 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=1860325 jiffies)
Mar 20 00:30:52 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 00:30:52 cheezum kernel: Call Trace:
Mar 20 00:30:52 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 00:30:52 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 00:30:52 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 00:30:52 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 00:30:52 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:30:52 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:30:52 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:30:52 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:30:52 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:30:52 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:30:52 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:30:52 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:30:52 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:30:52 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:30:52 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:30:52 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:30:52 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:30:52 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:30:52 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:30:52 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:30:52 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 00:30:52 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 00:30:52 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 00:30:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=1860300 jiffies)
Mar 20 00:30:53 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 00:30:53 cheezum kernel: Call Trace:
Mar 20 00:30:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 00:30:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 00:30:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:30:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:30:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:30:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:30:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:30:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:30:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:30:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:30:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:30:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:30:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:30:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:30:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:30:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:30:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:30:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 00:30:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 00:30:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:30:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 00:30:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 00:30:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 00:30:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 00:30:53 cheezum kernel:  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 00:30:53 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 00:30:53 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 00:30:53 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 00:30:53 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 00:30:53 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 00:30:53 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 00:30:53 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 00:30:53 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 00:30:53 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 00:30:53 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 00:33:52 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=2040358 jiffies)
Mar 20 00:33:52 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 00:33:52 cheezum kernel: Call Trace:
Mar 20 00:33:52 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 00:33:52 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 00:33:52 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 00:33:52 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 00:33:52 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:33:52 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:33:52 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:33:52 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:33:52 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:33:52 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:33:52 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:33:52 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:33:52 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:33:52 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:33:52 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:33:52 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:33:52 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:33:52 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:33:52 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:33:52 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:33:52 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 00:33:52 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 00:33:52 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 00:33:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 0 (t=2040330 jiffies)
Mar 20 00:33:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=2040330 jiffies)
Mar 20 00:33:53 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 00:33:53 cheezum kernel: Call Trace:
Mar 20 00:33:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 00:33:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 00:33:53 cheezum kernel: Pid: 0, comm: swapper/0 Not tainted 3.2.1-linode40 #1
Mar 20 00:33:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:33:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:33:53 cheezum kernel: Call Trace:
Mar 20 00:33:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:33:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:33:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:33:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 00:33:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:33:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:33:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:33:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 00:33:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:33:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:33:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:33:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:33:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:33:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:33:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:33:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:33:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:33:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:33:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:33:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 00:33:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 00:33:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:33:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:33:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 00:33:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:33:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 00:33:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 00:33:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:33:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 00:33:53 cheezum kernel:  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 00:33:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:33:53 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 00:33:53 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 00:33:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:33:53 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 00:33:53 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 00:33:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:33:53 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 00:33:53 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 00:33:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:33:53 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 00:33:53 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 00:33:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:33:53 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 00:33:53 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 00:33:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:33:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:33:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:33:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:33:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 00:33:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 00:33:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:33:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 00:33:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 00:33:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 00:33:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 00:33:53 cheezum kernel:  [<c05de8ba>] ? rt_garbage_collect+0x12a/0x400
Mar 20 00:33:53 cheezum kernel:  [<c0564add>] ? dst_alloc+0xfd/0x140
Mar 20 00:33:53 cheezum kernel:  [<c05dfc97>] ? rt_dst_alloc+0x37/0x40
Mar 20 00:33:53 cheezum kernel:  [<c05e16cf>] ? ip_route_input_slow+0x28f/0x560
Mar 20 00:33:53 cheezum kernel:  [<c05e0010>] ? __mkroute_output+0x10/0x2e0
Mar 20 00:33:53 cheezum kernel:  [<c059a2e4>] ? nf_hook_slow+0x64/0x130
Mar 20 00:33:53 cheezum kernel:  [<c05e3213>] ? ip_rcv_finish+0x243/0x380
Mar 20 00:33:53 cheezum kernel:  [<c055bad8>] ? __netif_receive_skb+0x2e8/0x350
Mar 20 00:33:53 cheezum kernel:  [<c055d3f7>] ? netif_receive_skb+0x67/0x70
Mar 20 00:33:53 cheezum kernel:  [<c04fcdb1>] ? handle_incoming_queue+0x1a1/0x270
Mar 20 00:33:53 cheezum kernel:  [<c04fd0a4>] ? xennet_poll+0x224/0x570
Mar 20 00:33:53 cheezum kernel:  [<c055db8a>] ? net_rx_action+0xea/0x190
Mar 20 00:33:53 cheezum kernel:  [<c013ec5c>] ? __do_softirq+0x7c/0x110
Mar 20 00:33:53 cheezum kernel:  [<c013ebe0>] ? irq_enter+0x60/0x60
Mar 20 00:33:53 cheezum kernel:  <IRQ>  [<c013ea3e>] ? irq_exit+0x6e/0xa0
Mar 20 00:33:53 cheezum kernel:  [<c048d9dd>] ? xen_evtchn_do_upcall+0x1d/0x30
Mar 20 00:33:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:33:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:33:53 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 00:33:53 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 00:33:53 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 00:33:53 cheezum kernel:  [<c08837d1>] ? start_kernel+0x2dd/0x2e2
Mar 20 00:33:53 cheezum kernel:  [<c0883332>] ? kernel_init+0x141/0x141
Mar 20 00:33:53 cheezum kernel:  [<c088672b>] ? xen_start_kernel+0x4f7/0x4ff
Mar 20 00:35:01 cheezum /USR/SBIN/CRON[12726]: (root) CMD ([ -x /usr/lib/sysstat/sa1 ] && { [ -r "$DEFAULT" ] && . "$DEFAULT" ; [ "$ENABLED" = "true" ] && exec /usr/lib/sysstat/sa1 $SA1_OPTIONS 1 1 ; })
Mar 20 00:36:52 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=2220391 jiffies)
Mar 20 00:36:52 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 00:36:52 cheezum kernel: Call Trace:
Mar 20 00:36:52 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 00:36:52 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 00:36:52 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 00:36:52 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 00:36:52 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:36:52 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:36:52 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:36:52 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:36:52 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:36:52 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:36:52 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:36:52 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:36:52 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:36:52 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:36:52 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:36:52 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:36:52 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:36:52 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:36:52 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:36:52 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:36:52 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 00:36:52 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 00:36:52 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 00:36:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=2220360 jiffies)
Mar 20 00:36:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 0 (t=2220360 jiffies)
Mar 20 00:36:53 cheezum kernel: Pid: 0, comm: swapper/0 Not tainted 3.2.1-linode40 #1
Mar 20 00:36:53 cheezum kernel: Call Trace:
Mar 20 00:36:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 00:36:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 00:36:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:36:53 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 00:36:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:36:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:36:53 cheezum kernel: Call Trace:
Mar 20 00:36:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:36:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 00:36:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:36:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 00:36:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:36:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:36:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:36:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:36:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:36:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:36:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:36:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:36:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:36:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:36:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:36:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:36:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:36:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:36:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:36:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:36:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 00:36:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 00:36:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:36:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:36:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 00:36:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:36:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 00:36:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 00:36:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:36:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 00:36:53 cheezum kernel:  [<c05de8ba>] ? rt_garbage_collect+0x12a/0x400
Mar 20 00:36:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:36:53 cheezum kernel:  [<c0564add>] ? dst_alloc+0xfd/0x140
Mar 20 00:36:53 cheezum kernel:  [<c05dfc97>] ? rt_dst_alloc+0x37/0x40
Mar 20 00:36:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:36:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:36:53 cheezum kernel:  [<c05e16cf>] ? ip_route_input_slow+0x28f/0x560
Mar 20 00:36:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:36:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:36:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:36:53 cheezum kernel:  [<c05e0010>] ? __mkroute_output+0x10/0x2e0
Mar 20 00:36:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:36:53 cheezum kernel:  [<c059a2e4>] ? nf_hook_slow+0x64/0x130
Mar 20 00:36:53 cheezum kernel:  [<c05e3213>] ? ip_rcv_finish+0x243/0x380
Mar 20 00:36:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:36:53 cheezum kernel:  [<c055bad8>] ? __netif_receive_skb+0x2e8/0x350
Mar 20 00:36:53 cheezum kernel:  [<c055d3f7>] ? netif_receive_skb+0x67/0x70
Mar 20 00:36:53 cheezum kernel:  [<c04fcdb1>] ? handle_incoming_queue+0x1a1/0x270
Mar 20 00:36:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 00:36:53 cheezum kernel:  [<c04fd0a4>] ? xennet_poll+0x224/0x570
Mar 20 00:36:53 cheezum kernel:  [<c055db8a>] ? net_rx_action+0xea/0x190
Mar 20 00:36:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 00:36:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:36:53 cheezum kernel:  [<c013ec5c>] ? __do_softirq+0x7c/0x110
Mar 20 00:36:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 00:36:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 00:36:53 cheezum kernel:  [<c013ebe0>] ? irq_enter+0x60/0x60
Mar 20 00:36:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 00:36:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 00:36:53 cheezum kernel:  <IRQ>  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 00:36:53 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 00:36:53 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 00:36:53 cheezum kernel:  [<c013ea3e>] ? irq_exit+0x6e/0xa0
Mar 20 00:36:53 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 00:36:53 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 00:36:53 cheezum kernel:  [<c048d9dd>] ? xen_evtchn_do_upcall+0x1d/0x30
Mar 20 00:36:53 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 00:36:53 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 00:36:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:36:53 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 00:36:53 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 00:36:53 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 00:36:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:36:53 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 00:36:53 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 00:36:53 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 00:36:53 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 00:36:53 cheezum kernel:  [<c08837d1>] ? start_kernel+0x2dd/0x2e2
Mar 20 00:36:53 cheezum kernel:  [<c0883332>] ? kernel_init+0x141/0x141
Mar 20 00:36:53 cheezum kernel:  [<c088672b>] ? xen_start_kernel+0x4f7/0x4ff
Mar 20 00:39:52 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=2400422 jiffies)
Mar 20 00:39:52 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 00:39:52 cheezum kernel: Call Trace:
Mar 20 00:39:52 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 00:39:52 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 00:39:52 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 00:39:52 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 00:39:52 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:39:52 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:39:52 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:39:52 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:39:52 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:39:52 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:39:52 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:39:52 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:39:52 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:39:52 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:39:52 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:39:52 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:39:52 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:39:52 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:39:52 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:39:52 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:39:52 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 00:39:52 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 00:39:52 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 00:39:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=2400390 jiffies)
Mar 20 00:39:53 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 00:39:53 cheezum kernel: Call Trace:
Mar 20 00:39:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 00:39:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 00:39:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:39:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:39:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:39:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:39:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:39:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:39:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:39:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:39:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:39:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:39:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:39:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:39:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:39:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:39:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:39:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:39:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 00:39:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 00:39:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:39:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 00:39:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 00:39:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 00:39:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 00:39:53 cheezum kernel:  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 00:39:53 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 00:39:53 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 00:39:53 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 00:39:53 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 00:39:53 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 00:39:53 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 00:39:53 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 00:39:53 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 00:39:53 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 00:39:53 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 00:42:52 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=2580454 jiffies)
Mar 20 00:42:52 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 00:42:52 cheezum kernel: Call Trace:
Mar 20 00:42:52 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 00:42:52 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 00:42:52 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 00:42:52 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 00:42:52 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:42:52 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:42:52 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:42:52 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:42:52 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:42:52 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:42:52 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:42:52 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:42:52 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:42:52 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:42:52 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:42:52 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:42:52 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:42:52 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:42:52 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:42:52 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:42:52 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 00:42:52 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 00:42:52 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 00:42:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=2580420 jiffies)
Mar 20 00:42:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 0 (t=2580420 jiffies)
Mar 20 00:42:53 cheezum kernel: Pid: 0, comm: swapper/0 Not tainted 3.2.1-linode40 #1
Mar 20 00:42:53 cheezum kernel: Call Trace:
Mar 20 00:42:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 00:42:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 00:42:53 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 00:42:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:42:53 cheezum kernel: Call Trace:
Mar 20 00:42:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:42:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:42:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:42:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:42:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 00:42:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:42:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:42:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 00:42:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:42:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:42:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:42:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:42:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:42:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:42:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:42:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:42:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:42:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:42:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:42:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:42:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:42:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:42:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:42:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 00:42:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 00:42:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:42:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:42:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 00:42:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:42:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 00:42:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:42:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 00:42:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 00:42:53 cheezum kernel:  [<c05de8ba>] ? rt_garbage_collect+0x12a/0x400
Mar 20 00:42:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:42:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:42:53 cheezum kernel:  [<c0564add>] ? dst_alloc+0xfd/0x140
Mar 20 00:42:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:42:53 cheezum kernel:  [<c05dfc97>] ? rt_dst_alloc+0x37/0x40
Mar 20 00:42:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:42:53 cheezum kernel:  [<c05e16cf>] ? ip_route_input_slow+0x28f/0x560
Mar 20 00:42:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:42:53 cheezum kernel:  [<c05e0010>] ? __mkroute_output+0x10/0x2e0
Mar 20 00:42:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:42:53 cheezum kernel:  [<c059a2e4>] ? nf_hook_slow+0x64/0x130
Mar 20 00:42:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 00:42:53 cheezum kernel:  [<c05e3213>] ? ip_rcv_finish+0x243/0x380
Mar 20 00:42:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 00:42:53 cheezum kernel:  [<c055bad8>] ? __netif_receive_skb+0x2e8/0x350
Mar 20 00:42:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:42:53 cheezum kernel:  [<c055d3f7>] ? netif_receive_skb+0x67/0x70
Mar 20 00:42:53 cheezum kernel:  [<c04fcdb1>] ? handle_incoming_queue+0x1a1/0x270
Mar 20 00:42:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 00:42:53 cheezum kernel:  [<c04fd0a4>] ? xennet_poll+0x224/0x570
Mar 20 00:42:53 cheezum kernel:  [<c055db8a>] ? net_rx_action+0xea/0x190
Mar 20 00:42:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 00:42:53 cheezum kernel:  [<c013ec5c>] ? __do_softirq+0x7c/0x110
Mar 20 00:42:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 00:42:53 cheezum kernel:  [<c013ebe0>] ? irq_enter+0x60/0x60
Mar 20 00:42:53 cheezum kernel:  <IRQ>  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 00:42:53 cheezum kernel:  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 00:42:53 cheezum kernel:  [<c013ea3e>] ? irq_exit+0x6e/0xa0
Mar 20 00:42:53 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 00:42:53 cheezum kernel:  [<c048d9dd>] ? xen_evtchn_do_upcall+0x1d/0x30
Mar 20 00:42:53 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 00:42:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:42:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:42:53 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 00:42:53 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 00:42:53 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 00:42:53 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 00:42:53 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 00:42:53 cheezum kernel:  [<c08837d1>] ? start_kernel+0x2dd/0x2e2
Mar 20 00:42:53 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 00:42:53 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 00:42:53 cheezum kernel:  [<c0883332>] ? kernel_init+0x141/0x141
Mar 20 00:42:53 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 00:42:53 cheezum kernel:  [<c088672b>] ? xen_start_kernel+0x4f7/0x4ff
Mar 20 00:42:53 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 00:42:53 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 00:42:53 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 00:45:01 cheezum /USR/SBIN/CRON[12739]: (root) CMD ([ -x /usr/lib/sysstat/sa1 ] && { [ -r "$DEFAULT" ] && . "$DEFAULT" ; [ "$ENABLED" = "true" ] && exec /usr/lib/sysstat/sa1 $SA1_OPTIONS 1 1 ; })
Mar 20 00:45:52 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=2760486 jiffies)
Mar 20 00:45:52 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 00:45:52 cheezum kernel: Call Trace:
Mar 20 00:45:52 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 00:45:52 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 00:45:52 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 00:45:52 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 00:45:52 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:45:52 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:45:52 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:45:52 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:45:52 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:45:52 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:45:52 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:45:52 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:45:52 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:45:52 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:45:52 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:45:52 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:45:52 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:45:52 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:45:52 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:45:52 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:45:52 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 00:45:52 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 00:45:52 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 00:45:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=2760450 jiffies)
Mar 20 00:45:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 0 (t=2760450 jiffies)
Mar 20 00:45:53 cheezum kernel: Pid: 0, comm: swapper/0 Not tainted 3.2.1-linode40 #1
Mar 20 00:45:53 cheezum kernel: Call Trace:
Mar 20 00:45:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 00:45:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 00:45:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:45:53 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 00:45:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:45:53 cheezum kernel: Call Trace:
Mar 20 00:45:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:45:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 00:45:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:45:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 00:45:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:45:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:45:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:45:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:45:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:45:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:45:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:45:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:45:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:45:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:45:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:45:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:45:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:45:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:45:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:45:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:45:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:45:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:45:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:45:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:45:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:45:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:45:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:45:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 00:45:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:45:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 00:45:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:45:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:45:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:45:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 00:45:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:45:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 00:45:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 00:45:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 00:45:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:45:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 00:45:53 cheezum kernel:  [<c05de8ba>] ? rt_garbage_collect+0x12a/0x400
Mar 20 00:45:53 cheezum kernel:  [<c0564add>] ? dst_alloc+0xfd/0x140
Mar 20 00:45:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 00:45:53 cheezum kernel:  [<c05dfc97>] ? rt_dst_alloc+0x37/0x40
Mar 20 00:45:53 cheezum kernel:  [<c05e16cf>] ? ip_route_input_slow+0x28f/0x560
Mar 20 00:45:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:45:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 00:45:53 cheezum kernel:  [<c05e0010>] ? __mkroute_output+0x10/0x2e0
Mar 20 00:45:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 00:45:53 cheezum kernel:  [<c059a2e4>] ? nf_hook_slow+0x64/0x130
Mar 20 00:45:53 cheezum kernel:  [<c05e3213>] ? ip_rcv_finish+0x243/0x380
Mar 20 00:45:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 00:45:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 00:45:53 cheezum kernel:  [<c055bad8>] ? __netif_receive_skb+0x2e8/0x350
Mar 20 00:45:53 cheezum kernel:  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 00:45:53 cheezum kernel:  [<c055d3f7>] ? netif_receive_skb+0x67/0x70
Mar 20 00:45:53 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 00:45:53 cheezum kernel:  [<c04fcdb1>] ? handle_incoming_queue+0x1a1/0x270
Mar 20 00:45:53 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 00:45:53 cheezum kernel:  [<c04fd0a4>] ? xennet_poll+0x224/0x570
Mar 20 00:45:53 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 00:45:53 cheezum kernel:  [<c055db8a>] ? net_rx_action+0xea/0x190
Mar 20 00:45:53 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 00:45:53 cheezum kernel:  [<c013ec5c>] ? __do_softirq+0x7c/0x110
Mar 20 00:45:53 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 00:45:53 cheezum kernel:  [<c013ebe0>] ? irq_enter+0x60/0x60
Mar 20 00:45:53 cheezum kernel:  <IRQ>  [<c013ea3e>] ? irq_exit+0x6e/0xa0
Mar 20 00:45:53 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 00:45:53 cheezum kernel:  [<c048d9dd>] ? xen_evtchn_do_upcall+0x1d/0x30
Mar 20 00:45:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:45:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:45:53 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 00:45:53 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 00:45:53 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 00:45:53 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 00:45:53 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 00:45:53 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 00:45:53 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 00:45:53 cheezum kernel:  [<c08837d1>] ? start_kernel+0x2dd/0x2e2
Mar 20 00:45:53 cheezum kernel:  [<c0883332>] ? kernel_init+0x141/0x141
Mar 20 00:45:53 cheezum kernel:  [<c088672b>] ? xen_start_kernel+0x4f7/0x4ff
Mar 20 00:48:53 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=2940519 jiffies)
Mar 20 00:48:53 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 00:48:53 cheezum kernel: Call Trace:
Mar 20 00:48:53 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 00:48:53 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 00:48:53 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 00:48:53 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 00:48:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:48:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:48:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:48:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:48:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:48:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:48:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:48:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:48:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:48:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:48:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:48:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:48:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:48:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:48:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:48:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:48:53 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 00:48:53 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 00:48:53 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 00:48:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=2940480 jiffies)
Mar 20 00:48:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 0 (t=2940480 jiffies)
Mar 20 00:48:53 cheezum kernel: Pid: 0, comm: swapper/0 Not tainted 3.2.1-linode40 #1
Mar 20 00:48:53 cheezum kernel: Call Trace:
Mar 20 00:48:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 00:48:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 00:48:53 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 00:48:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:48:53 cheezum kernel: Call Trace:
Mar 20 00:48:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:48:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:48:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 00:48:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:48:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:48:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 00:48:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:48:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:48:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:48:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:48:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:48:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:48:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:48:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:48:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:48:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:48:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:48:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:48:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:48:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:48:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:48:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:48:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:48:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:48:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:48:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:48:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:48:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:48:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:48:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:48:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:48:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:48:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 00:48:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:48:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 00:48:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 00:48:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 00:48:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:48:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:48:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 00:48:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 00:48:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 00:48:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 00:48:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 00:48:53 cheezum kernel:  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 00:48:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 00:48:53 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 00:48:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 00:48:53 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 00:48:53 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 00:48:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 00:48:53 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 00:48:53 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 00:48:53 cheezum kernel:  [<c05de8ba>] ? rt_garbage_collect+0x12a/0x400
Mar 20 00:48:53 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 00:48:53 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 00:48:53 cheezum kernel:  [<c0564add>] ? dst_alloc+0xfd/0x140
Mar 20 00:48:53 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 00:48:53 cheezum kernel:  [<c05dfc97>] ? rt_dst_alloc+0x37/0x40
Mar 20 00:48:53 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 00:48:53 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 00:48:53 cheezum kernel:  [<c05e16cf>] ? ip_route_input_slow+0x28f/0x560
Mar 20 00:48:53 cheezum kernel:  [<c05e0010>] ? __mkroute_output+0x10/0x2e0
Mar 20 00:48:53 cheezum kernel:  [<c059a2e4>] ? nf_hook_slow+0x64/0x130
Mar 20 00:48:53 cheezum kernel:  [<c05e3213>] ? ip_rcv_finish+0x243/0x380
Mar 20 00:48:53 cheezum kernel:  [<c055bad8>] ? __netif_receive_skb+0x2e8/0x350
Mar 20 00:48:53 cheezum kernel:  [<c055d3f7>] ? netif_receive_skb+0x67/0x70
Mar 20 00:48:53 cheezum kernel:  [<c04fcdb1>] ? handle_incoming_queue+0x1a1/0x270
Mar 20 00:48:53 cheezum kernel:  [<c04fd0a4>] ? xennet_poll+0x224/0x570
Mar 20 00:48:53 cheezum kernel:  [<c055db8a>] ? net_rx_action+0xea/0x190
Mar 20 00:48:53 cheezum kernel:  [<c013ec5c>] ? __do_softirq+0x7c/0x110
Mar 20 00:48:53 cheezum kernel:  [<c013ebe0>] ? irq_enter+0x60/0x60
Mar 20 00:48:53 cheezum kernel:  <IRQ>  [<c013ea3e>] ? irq_exit+0x6e/0xa0
Mar 20 00:48:53 cheezum kernel:  [<c048d9dd>] ? xen_evtchn_do_upcall+0x1d/0x30
Mar 20 00:48:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:48:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:48:53 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 00:48:53 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 00:48:53 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 00:48:53 cheezum kernel:  [<c08837d1>] ? start_kernel+0x2dd/0x2e2
Mar 20 00:48:53 cheezum kernel:  [<c0883332>] ? kernel_init+0x141/0x141
Mar 20 00:48:53 cheezum kernel:  [<c088672b>] ? xen_start_kernel+0x4f7/0x4ff
Mar 20 00:51:53 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=3120550 jiffies)
Mar 20 00:51:53 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 00:51:53 cheezum kernel: Call Trace:
Mar 20 00:51:53 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 00:51:53 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 00:51:53 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 00:51:53 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 00:51:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:51:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:51:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:51:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:51:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:51:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:51:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:51:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:51:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:51:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:51:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:51:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:51:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:51:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:51:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:51:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:51:53 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 00:51:53 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 00:51:53 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 00:51:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=3120510 jiffies)
Mar 20 00:51:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 0 (t=3120510 jiffies)
Mar 20 00:51:53 cheezum kernel: Pid: 0, comm: swapper/0 Not tainted 3.2.1-linode40 #1
Mar 20 00:51:53 cheezum kernel: Call Trace:
Mar 20 00:51:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 00:51:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 00:51:53 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 00:51:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:51:53 cheezum kernel: Call Trace:
Mar 20 00:51:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:51:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 00:51:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 00:51:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:51:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:51:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:51:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:51:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:51:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:51:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:51:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:51:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:51:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:51:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:51:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:51:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:51:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:51:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:51:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:51:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:51:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:51:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:51:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:51:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:51:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:51:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:51:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:51:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:51:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 00:51:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:51:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 00:51:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:51:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 00:51:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:51:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:51:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 00:51:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:51:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 00:51:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 00:51:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 00:51:53 cheezum kernel:  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 00:51:53 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 00:51:53 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 00:51:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 00:51:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:51:53 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 00:51:53 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 00:51:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 00:51:53 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 00:51:53 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 00:51:53 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 00:51:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 00:51:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 00:51:53 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 00:51:53 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 00:51:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 00:51:53 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 00:51:53 cheezum kernel:  [<c05de8ba>] ? rt_garbage_collect+0x12a/0x400
Mar 20 00:51:53 cheezum kernel:  [<c0564add>] ? dst_alloc+0xfd/0x140
Mar 20 00:51:53 cheezum kernel:  [<c05dfc97>] ? rt_dst_alloc+0x37/0x40
Mar 20 00:51:53 cheezum kernel:  [<c05e16cf>] ? ip_route_input_slow+0x28f/0x560
Mar 20 00:51:53 cheezum kernel:  [<c05e0010>] ? __mkroute_output+0x10/0x2e0
Mar 20 00:51:53 cheezum kernel:  [<c059a2e4>] ? nf_hook_slow+0x64/0x130
Mar 20 00:51:53 cheezum kernel:  [<c05e3213>] ? ip_rcv_finish+0x243/0x380
Mar 20 00:51:53 cheezum kernel:  [<c055bad8>] ? __netif_receive_skb+0x2e8/0x350
Mar 20 00:51:53 cheezum kernel:  [<c055d3f7>] ? netif_receive_skb+0x67/0x70
Mar 20 00:51:53 cheezum kernel:  [<c04fcdb1>] ? handle_incoming_queue+0x1a1/0x270
Mar 20 00:51:53 cheezum kernel:  [<c04fd0a4>] ? xennet_poll+0x224/0x570
Mar 20 00:51:53 cheezum kernel:  [<c055db8a>] ? net_rx_action+0xea/0x190
Mar 20 00:51:53 cheezum kernel:  [<c013ec5c>] ? __do_softirq+0x7c/0x110
Mar 20 00:51:53 cheezum kernel:  [<c013ebe0>] ? irq_enter+0x60/0x60
Mar 20 00:51:53 cheezum kernel:  <IRQ>  [<c013ea3e>] ? irq_exit+0x6e/0xa0
Mar 20 00:51:53 cheezum kernel:  [<c048d9dd>] ? xen_evtchn_do_upcall+0x1d/0x30
Mar 20 00:51:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:51:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:51:53 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 00:51:53 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 00:51:53 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 00:51:53 cheezum kernel:  [<c08837d1>] ? start_kernel+0x2dd/0x2e2
Mar 20 00:51:53 cheezum kernel:  [<c0883332>] ? kernel_init+0x141/0x141
Mar 20 00:51:53 cheezum kernel:  [<c088672b>] ? xen_start_kernel+0x4f7/0x4ff
Mar 20 00:54:53 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=3300583 jiffies)
Mar 20 00:54:53 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 00:54:53 cheezum kernel: Call Trace:
Mar 20 00:54:53 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 00:54:53 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 00:54:53 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 00:54:53 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 00:54:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:54:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:54:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:54:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:54:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:54:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:54:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:54:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:54:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:54:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:54:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:54:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:54:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:54:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:54:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:54:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:54:53 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 00:54:53 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 00:54:53 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 00:54:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 0 (t=3300540 jiffies)
Mar 20 00:54:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=3300540 jiffies)
Mar 20 00:54:53 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 00:54:53 cheezum kernel: Call Trace:
Mar 20 00:54:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 00:54:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 00:54:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:54:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:54:53 cheezum kernel: Pid: 0, comm: swapper/0 Not tainted 3.2.1-linode40 #1
Mar 20 00:54:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:54:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:54:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:54:53 cheezum kernel: Call Trace:
Mar 20 00:54:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:54:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 00:54:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:54:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:54:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 00:54:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:54:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:54:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:54:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:54:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:54:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:54:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:54:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:54:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:54:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:54:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:54:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:54:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 00:54:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 00:54:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:54:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:54:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 00:54:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:54:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 00:54:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 00:54:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:54:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 00:54:53 cheezum kernel:  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 00:54:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:54:53 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 00:54:53 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 00:54:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:54:53 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 00:54:53 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 00:54:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:54:53 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 00:54:53 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 00:54:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:54:53 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 00:54:53 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 00:54:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:54:53 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 00:54:53 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 00:54:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:54:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:54:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:54:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:54:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 00:54:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 00:54:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:54:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 00:54:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 00:54:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 00:54:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 00:54:53 cheezum kernel:  [<c05de8ba>] ? rt_garbage_collect+0x12a/0x400
Mar 20 00:54:53 cheezum kernel:  [<c0564add>] ? dst_alloc+0xfd/0x140
Mar 20 00:54:53 cheezum kernel:  [<c05dfc97>] ? rt_dst_alloc+0x37/0x40
Mar 20 00:54:53 cheezum kernel:  [<c05e16cf>] ? ip_route_input_slow+0x28f/0x560
Mar 20 00:54:53 cheezum kernel:  [<c05e0010>] ? __mkroute_output+0x10/0x2e0
Mar 20 00:54:53 cheezum kernel:  [<c059a2e4>] ? nf_hook_slow+0x64/0x130
Mar 20 00:54:53 cheezum kernel:  [<c05e3213>] ? ip_rcv_finish+0x243/0x380
Mar 20 00:54:53 cheezum kernel:  [<c055bad8>] ? __netif_receive_skb+0x2e8/0x350
Mar 20 00:54:53 cheezum kernel:  [<c055d3f7>] ? netif_receive_skb+0x67/0x70
Mar 20 00:54:53 cheezum kernel:  [<c04fcdb1>] ? handle_incoming_queue+0x1a1/0x270
Mar 20 00:54:53 cheezum kernel:  [<c04fd0a4>] ? xennet_poll+0x224/0x570
Mar 20 00:54:53 cheezum kernel:  [<c055db8a>] ? net_rx_action+0xea/0x190
Mar 20 00:54:53 cheezum kernel:  [<c013ec5c>] ? __do_softirq+0x7c/0x110
Mar 20 00:54:53 cheezum kernel:  [<c013ebe0>] ? irq_enter+0x60/0x60
Mar 20 00:54:53 cheezum kernel:  <IRQ>  [<c013ea3e>] ? irq_exit+0x6e/0xa0
Mar 20 00:54:53 cheezum kernel:  [<c048d9dd>] ? xen_evtchn_do_upcall+0x1d/0x30
Mar 20 00:54:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:54:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:54:53 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 00:54:53 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 00:54:53 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 00:54:53 cheezum kernel:  [<c08837d1>] ? start_kernel+0x2dd/0x2e2
Mar 20 00:54:53 cheezum kernel:  [<c0883332>] ? kernel_init+0x141/0x141
Mar 20 00:54:53 cheezum kernel:  [<c088672b>] ? xen_start_kernel+0x4f7/0x4ff
Mar 20 00:55:01 cheezum /USR/SBIN/CRON[12761]: (root) CMD ([ -x /usr/lib/sysstat/sa1 ] && { [ -r "$DEFAULT" ] && . "$DEFAULT" ; [ "$ENABLED" = "true" ] && exec /usr/lib/sysstat/sa1 $SA1_OPTIONS 1 1 ; })
Mar 20 00:57:53 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=3480615 jiffies)
Mar 20 00:57:53 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 00:57:53 cheezum kernel: Call Trace:
Mar 20 00:57:53 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 00:57:53 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 00:57:53 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 00:57:53 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 00:57:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:57:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:57:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:57:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:57:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:57:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:57:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:57:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:57:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:57:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:57:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:57:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:57:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:57:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:57:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:57:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:57:53 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 00:57:53 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 00:57:53 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 00:57:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=3480570 jiffies)
Mar 20 00:57:53 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 00:57:53 cheezum kernel: Call Trace:
Mar 20 00:57:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 00:57:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 00:57:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 00:57:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 00:57:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 00:57:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 00:57:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 00:57:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 00:57:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 00:57:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 00:57:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 00:57:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 00:57:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 00:57:53 cheezum kernel:  [<c0181012>] ? handle_irq_event_percpu+0x72/0x140
Mar 20 00:57:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 00:57:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 00:57:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 00:57:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 00:57:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 00:57:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 00:57:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 00:57:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 00:57:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 00:57:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 00:57:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 00:57:53 cheezum kernel:  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 00:57:53 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 00:57:53 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 00:57:53 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 00:57:53 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 00:57:53 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 00:57:53 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 00:57:53 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 00:57:53 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 00:57:53 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 00:57:53 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 01:00:53 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=3660648 jiffies)
Mar 20 01:00:53 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 01:00:53 cheezum kernel: Call Trace:
Mar 20 01:00:53 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 01:00:53 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 01:00:53 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 01:00:53 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 01:00:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:00:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:00:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:00:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:00:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:00:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:00:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:00:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:00:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:00:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:00:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:00:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:00:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:00:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:00:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:00:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:00:53 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 01:00:53 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 01:00:53 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 01:00:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=3660600 jiffies)
Mar 20 01:00:53 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 01:00:53 cheezum kernel: Call Trace:
Mar 20 01:00:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 01:00:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 01:00:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:00:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:00:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:00:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:00:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:00:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:00:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:00:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:00:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:00:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:00:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:00:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:00:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:00:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:00:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:00:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:00:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 01:00:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 01:00:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:00:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 01:00:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 01:00:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 01:00:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 01:00:53 cheezum kernel:  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 01:00:53 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 01:00:53 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 01:00:53 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 01:00:53 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 01:00:53 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 01:00:53 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 01:00:53 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 01:00:53 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 01:00:53 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 01:00:53 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 01:03:53 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=3840680 jiffies)
Mar 20 01:03:53 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 01:03:53 cheezum kernel: Call Trace:
Mar 20 01:03:53 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 01:03:53 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 01:03:53 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 01:03:53 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 01:03:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:03:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:03:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:03:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:03:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:03:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:03:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:03:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:03:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:03:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:03:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:03:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:03:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:03:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:03:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:03:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:03:53 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 01:03:53 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 01:03:53 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 01:03:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=3840630 jiffies)
Mar 20 01:03:53 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 01:03:53 cheezum kernel: Call Trace:
Mar 20 01:03:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 01:03:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 01:03:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:03:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:03:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:03:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:03:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:03:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:03:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:03:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:03:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:03:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:03:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:03:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:03:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:03:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:03:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:03:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:03:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 01:03:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 01:03:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:03:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 01:03:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 01:03:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 01:03:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 01:03:53 cheezum kernel:  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 01:03:53 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 01:03:53 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 01:03:53 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 01:03:53 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 01:03:53 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 01:03:53 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 01:03:53 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 01:03:53 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 01:03:53 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 01:03:53 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 01:05:01 cheezum /USR/SBIN/CRON[12776]: (root) CMD ([ -x /usr/lib/sysstat/sa1 ] && { [ -r "$DEFAULT" ] && . "$DEFAULT" ; [ "$ENABLED" = "true" ] && exec /usr/lib/sysstat/sa1 $SA1_OPTIONS 1 1 ; })
Mar 20 01:06:53 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=4020713 jiffies)
Mar 20 01:06:53 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 01:06:53 cheezum kernel: Call Trace:
Mar 20 01:06:53 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 01:06:53 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 01:06:53 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 01:06:53 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 01:06:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:06:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:06:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:06:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:06:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:06:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:06:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:06:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:06:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:06:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:06:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:06:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:06:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:06:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:06:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:06:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:06:53 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 01:06:53 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 01:06:53 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 01:06:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=4020660 jiffies)
Mar 20 01:06:53 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 01:06:53 cheezum kernel: Call Trace:
Mar 20 01:06:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 01:06:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 01:06:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:06:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:06:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:06:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:06:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:06:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:06:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:06:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:06:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:06:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:06:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:06:53 cheezum kernel:  [<c0181012>] ? handle_irq_event_percpu+0x72/0x140
Mar 20 01:06:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:06:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:06:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:06:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:06:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 01:06:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 01:06:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:06:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 01:06:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 01:06:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 01:06:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 01:06:53 cheezum kernel:  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 01:06:53 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 01:06:53 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 01:06:53 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 01:06:53 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 01:06:53 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 01:06:53 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 01:06:53 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 01:06:53 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 01:06:53 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 01:06:53 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 01:09:53 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=4200745 jiffies)
Mar 20 01:09:53 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 01:09:53 cheezum kernel: Call Trace:
Mar 20 01:09:53 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 01:09:53 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 01:09:53 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 01:09:53 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 01:09:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:09:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:09:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:09:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:09:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:09:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:09:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:09:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:09:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:09:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:09:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:09:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:09:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:09:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:09:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:09:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:09:53 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 01:09:53 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 01:09:53 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 01:09:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=4200690 jiffies)
Mar 20 01:09:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 0 (t=4200690 jiffies)
Mar 20 01:09:53 cheezum kernel: Pid: 0, comm: swapper/0 Not tainted 3.2.1-linode40 #1
Mar 20 01:09:53 cheezum kernel: Call Trace:
Mar 20 01:09:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 01:09:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 01:09:53 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 01:09:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:09:53 cheezum kernel: Call Trace:
Mar 20 01:09:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 01:09:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:09:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 01:09:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:09:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:09:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:09:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:09:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:09:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:09:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:09:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:09:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:09:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:09:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:09:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:09:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:09:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:09:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:09:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:09:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:09:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:09:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:09:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:09:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:09:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:09:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:09:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:09:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:09:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 01:09:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:09:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:09:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 01:09:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:09:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:09:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:09:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:09:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 01:09:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 01:09:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 01:09:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 01:09:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:09:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 01:09:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 01:09:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 01:09:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 01:09:53 cheezum kernel:  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 01:09:53 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 01:09:53 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 01:09:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 01:09:53 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 01:09:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 01:09:53 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 01:09:53 cheezum kernel:  [<c05de8ba>] ? rt_garbage_collect+0x12a/0x400
Mar 20 01:09:53 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 01:09:53 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 01:09:53 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 01:09:53 cheezum kernel:  [<c0564add>] ? dst_alloc+0xfd/0x140
Mar 20 01:09:53 cheezum kernel:  [<c05dfc97>] ? rt_dst_alloc+0x37/0x40
Mar 20 01:09:53 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 01:09:53 cheezum kernel:  [<c05e16cf>] ? ip_route_input_slow+0x28f/0x560
Mar 20 01:09:53 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 01:09:53 cheezum kernel:  [<c05e0010>] ? __mkroute_output+0x10/0x2e0
Mar 20 01:09:53 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 01:09:53 cheezum kernel:  [<c059a2e4>] ? nf_hook_slow+0x64/0x130
Mar 20 01:09:53 cheezum kernel:  [<c05e3213>] ? ip_rcv_finish+0x243/0x380
Mar 20 01:09:53 cheezum kernel:  [<c055bad8>] ? __netif_receive_skb+0x2e8/0x350
Mar 20 01:09:53 cheezum kernel:  [<c055d3f7>] ? netif_receive_skb+0x67/0x70
Mar 20 01:09:53 cheezum kernel:  [<c04fcdb1>] ? handle_incoming_queue+0x1a1/0x270
Mar 20 01:09:53 cheezum kernel:  [<c04fd0a4>] ? xennet_poll+0x224/0x570
Mar 20 01:09:53 cheezum kernel:  [<c055db8a>] ? net_rx_action+0xea/0x190
Mar 20 01:09:53 cheezum kernel:  [<c013ec5c>] ? __do_softirq+0x7c/0x110
Mar 20 01:09:53 cheezum kernel:  [<c013ebe0>] ? irq_enter+0x60/0x60
Mar 20 01:09:53 cheezum kernel:  <IRQ>  [<c013ea3e>] ? irq_exit+0x6e/0xa0
Mar 20 01:09:53 cheezum kernel:  [<c048d9dd>] ? xen_evtchn_do_upcall+0x1d/0x30
Mar 20 01:09:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:09:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:09:53 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 01:09:53 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 01:09:53 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 01:09:53 cheezum kernel:  [<c08837d1>] ? start_kernel+0x2dd/0x2e2
Mar 20 01:09:53 cheezum kernel:  [<c0883332>] ? kernel_init+0x141/0x141
Mar 20 01:09:53 cheezum kernel:  [<c088672b>] ? xen_start_kernel+0x4f7/0x4ff
Mar 20 01:12:53 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=4380778 jiffies)
Mar 20 01:12:53 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 01:12:53 cheezum kernel: Call Trace:
Mar 20 01:12:53 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 01:12:53 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 01:12:53 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 01:12:53 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 01:12:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:12:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:12:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:12:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:12:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:12:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:12:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:12:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:12:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:12:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:12:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:12:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:12:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:12:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:12:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:12:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:12:53 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 01:12:53 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 01:12:53 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 01:12:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=4380720 jiffies)
Mar 20 01:12:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 0 (t=4380720 jiffies)
Mar 20 01:12:53 cheezum kernel: Pid: 0, comm: swapper/0 Not tainted 3.2.1-linode40 #1
Mar 20 01:12:53 cheezum kernel: Call Trace:
Mar 20 01:12:53 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 01:12:53 cheezum kernel: Call Trace:
Mar 20 01:12:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 01:12:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 01:12:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 01:12:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 01:12:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:12:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:12:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:12:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:12:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:12:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:12:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:12:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:12:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:12:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:12:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:12:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:12:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:12:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:12:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:12:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:12:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:12:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:12:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:12:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:12:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:12:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:12:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:12:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:12:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:12:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:12:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:12:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:12:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:12:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 01:12:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 01:12:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:12:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:12:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 01:12:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:12:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:12:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 01:12:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 01:12:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 01:12:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 01:12:53 cheezum kernel:  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 01:12:53 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 01:12:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 01:12:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:12:53 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 01:12:53 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 01:12:53 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 01:12:53 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 01:12:53 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 01:12:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 01:12:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 01:12:53 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 01:12:53 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 01:12:53 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 01:12:53 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 01:12:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 01:12:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 01:12:53 cheezum kernel:  [<c05de8ba>] ? rt_garbage_collect+0x12a/0x400
Mar 20 01:12:53 cheezum kernel:  [<c0564add>] ? dst_alloc+0xfd/0x140
Mar 20 01:12:53 cheezum kernel:  [<c05dfc97>] ? rt_dst_alloc+0x37/0x40
Mar 20 01:12:53 cheezum kernel:  [<c05e16cf>] ? ip_route_input_slow+0x28f/0x560
Mar 20 01:12:53 cheezum kernel:  [<c05e0010>] ? __mkroute_output+0x10/0x2e0
Mar 20 01:12:53 cheezum kernel:  [<c059a2e4>] ? nf_hook_slow+0x64/0x130
Mar 20 01:12:53 cheezum kernel:  [<c05e3213>] ? ip_rcv_finish+0x243/0x380
Mar 20 01:12:53 cheezum kernel:  [<c055bad8>] ? __netif_receive_skb+0x2e8/0x350
Mar 20 01:12:53 cheezum kernel:  [<c055d3f7>] ? netif_receive_skb+0x67/0x70
Mar 20 01:12:53 cheezum kernel:  [<c04fcdb1>] ? handle_incoming_queue+0x1a1/0x270
Mar 20 01:12:53 cheezum kernel:  [<c04fd0a4>] ? xennet_poll+0x224/0x570
Mar 20 01:12:53 cheezum kernel:  [<c055db8a>] ? net_rx_action+0xea/0x190
Mar 20 01:12:53 cheezum kernel:  [<c013ec5c>] ? __do_softirq+0x7c/0x110
Mar 20 01:12:53 cheezum kernel:  [<c013ebe0>] ? irq_enter+0x60/0x60
Mar 20 01:12:53 cheezum kernel:  <IRQ>  [<c013ea3e>] ? irq_exit+0x6e/0xa0
Mar 20 01:12:53 cheezum kernel:  [<c048d9dd>] ? xen_evtchn_do_upcall+0x1d/0x30
Mar 20 01:12:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:12:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:12:53 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 01:12:53 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 01:12:53 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 01:12:53 cheezum kernel:  [<c08837d1>] ? start_kernel+0x2dd/0x2e2
Mar 20 01:12:53 cheezum kernel:  [<c0883332>] ? kernel_init+0x141/0x141
Mar 20 01:12:53 cheezum kernel:  [<c088672b>] ? xen_start_kernel+0x4f7/0x4ff
Mar 20 01:15:01 cheezum /USR/SBIN/CRON[12795]: (root) CMD ([ -x /usr/lib/sysstat/sa1 ] && { [ -r "$DEFAULT" ] && . "$DEFAULT" ; [ "$ENABLED" = "true" ] && exec /usr/lib/sysstat/sa1 $SA1_OPTIONS 1 1 ; })
Mar 20 01:15:53 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=4560810 jiffies)
Mar 20 01:15:53 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 01:15:53 cheezum kernel: Call Trace:
Mar 20 01:15:53 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 01:15:53 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 01:15:53 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 01:15:53 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 01:15:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:15:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:15:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:15:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:15:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:15:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:15:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:15:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:15:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:15:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:15:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:15:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:15:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:15:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:15:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:15:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:15:53 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 01:15:53 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 01:15:53 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 01:15:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=4560750 jiffies)
Mar 20 01:15:53 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 01:15:53 cheezum kernel: Call Trace:
Mar 20 01:15:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 01:15:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 01:15:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:15:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:15:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:15:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:15:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:15:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:15:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:15:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:15:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:15:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:15:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:15:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:15:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:15:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:15:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:15:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:15:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 01:15:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 01:15:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:15:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 01:15:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 01:15:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 01:15:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 01:15:53 cheezum kernel:  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 01:15:53 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 01:15:53 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 01:15:53 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 01:15:53 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 01:15:53 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 01:15:53 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 01:15:53 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 01:15:53 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 01:15:53 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 01:15:53 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 01:17:01 cheezum /USR/SBIN/CRON[12802]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Mar 20 01:18:53 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=4740842 jiffies)
Mar 20 01:18:53 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 01:18:53 cheezum kernel: Call Trace:
Mar 20 01:18:53 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 01:18:53 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 01:18:53 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 01:18:53 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 01:18:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:18:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:18:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:18:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:18:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:18:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:18:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:18:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:18:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:18:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:18:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:18:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:18:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:18:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:18:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:18:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:18:53 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 01:18:53 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 01:18:53 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 01:18:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=4740780 jiffies)
Mar 20 01:18:53 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 01:18:53 cheezum kernel: Call Trace:
Mar 20 01:18:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 01:18:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 01:18:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:18:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:18:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:18:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:18:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:18:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:18:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:18:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:18:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:18:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:18:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:18:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:18:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:18:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:18:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:18:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:18:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 01:18:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 01:18:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:18:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 01:18:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 01:18:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 01:18:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 01:18:53 cheezum kernel:  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 01:18:53 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 01:18:53 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 01:18:53 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 01:18:53 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 01:18:53 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 01:18:53 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 01:18:53 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 01:18:53 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 01:18:53 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 01:18:53 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 01:21:53 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=4920875 jiffies)
Mar 20 01:21:53 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 01:21:53 cheezum kernel: Call Trace:
Mar 20 01:21:53 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 01:21:53 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 01:21:53 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 01:21:53 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 01:21:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:21:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:21:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:21:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:21:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:21:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:21:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:21:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:21:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:21:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:21:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:21:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:21:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:21:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:21:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:21:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:21:53 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 01:21:53 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 01:21:53 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 01:21:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=4920810 jiffies)
Mar 20 01:21:53 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 01:21:53 cheezum kernel: Call Trace:
Mar 20 01:21:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 01:21:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 01:21:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:21:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:21:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:21:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:21:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:21:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:21:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:21:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:21:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:21:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:21:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:21:53 cheezum kernel:  [<c0181012>] ? handle_irq_event_percpu+0x72/0x140
Mar 20 01:21:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:21:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:21:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:21:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:21:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 01:21:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 01:21:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:21:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 01:21:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 01:21:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 01:21:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 01:21:53 cheezum kernel:  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 01:21:53 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 01:21:53 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 01:21:53 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 01:21:53 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 01:21:53 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 01:21:53 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 01:21:53 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 01:21:53 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 01:21:53 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 01:21:53 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 01:24:53 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=5100907 jiffies)
Mar 20 01:24:53 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 01:24:53 cheezum kernel: Call Trace:
Mar 20 01:24:53 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 01:24:53 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 01:24:53 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 01:24:53 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 01:24:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:24:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:24:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:24:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:24:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:24:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:24:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:24:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:24:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:24:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:24:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:24:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:24:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:24:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:24:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:24:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:24:53 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 01:24:53 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 01:24:53 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 01:24:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=5100840 jiffies)
Mar 20 01:24:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 0 (t=5100840 jiffies)
Mar 20 01:24:53 cheezum kernel: Pid: 0, comm: swapper/0 Not tainted 3.2.1-linode40 #1
Mar 20 01:24:53 cheezum kernel: Call Trace:
Mar 20 01:24:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 01:24:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 01:24:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:24:53 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 01:24:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:24:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:24:53 cheezum kernel: Call Trace:
Mar 20 01:24:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:24:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:24:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:24:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 01:24:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:24:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:24:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:24:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 01:24:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:24:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:24:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:24:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:24:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:24:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:24:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:24:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:24:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:24:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:24:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 01:24:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:24:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:24:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 01:24:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:24:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 01:24:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:24:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:24:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 01:24:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:24:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:24:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 01:24:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:24:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 01:24:53 cheezum kernel:  [<c05de8ba>] ? rt_garbage_collect+0x12a/0x400
Mar 20 01:24:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:24:53 cheezum kernel:  [<c0564add>] ? dst_alloc+0xfd/0x140
Mar 20 01:24:53 cheezum kernel:  [<c05dfc97>] ? rt_dst_alloc+0x37/0x40
Mar 20 01:24:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:24:53 cheezum kernel:  [<c05e16cf>] ? ip_route_input_slow+0x28f/0x560
Mar 20 01:24:53 cheezum kernel:  [<c05e0010>] ? __mkroute_output+0x10/0x2e0
Mar 20 01:24:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:24:53 cheezum kernel:  [<c059a2e4>] ? nf_hook_slow+0x64/0x130
Mar 20 01:24:53 cheezum kernel:  [<c05e3213>] ? ip_rcv_finish+0x243/0x380
Mar 20 01:24:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:24:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:24:53 cheezum kernel:  [<c055bad8>] ? __netif_receive_skb+0x2e8/0x350
Mar 20 01:24:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 01:24:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 01:24:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:24:53 cheezum kernel:  [<c055d3f7>] ? netif_receive_skb+0x67/0x70
Mar 20 01:24:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 01:24:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 01:24:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 01:24:53 cheezum kernel:  [<c04fcdb1>] ? handle_incoming_queue+0x1a1/0x270
Mar 20 01:24:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 01:24:53 cheezum kernel:  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 01:24:53 cheezum kernel:  [<c04fd0a4>] ? xennet_poll+0x224/0x570
Mar 20 01:24:53 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 01:24:53 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 01:24:53 cheezum kernel:  [<c055db8a>] ? net_rx_action+0xea/0x190
Mar 20 01:24:53 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 01:24:53 cheezum kernel:  [<c013ec5c>] ? __do_softirq+0x7c/0x110
Mar 20 01:24:53 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 01:24:53 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 01:24:53 cheezum kernel:  [<c013ebe0>] ? irq_enter+0x60/0x60
Mar 20 01:24:53 cheezum kernel:  <IRQ>  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 01:24:53 cheezum kernel:  [<c013ea3e>] ? irq_exit+0x6e/0xa0
Mar 20 01:24:53 cheezum kernel:  [<c048d9dd>] ? xen_evtchn_do_upcall+0x1d/0x30
Mar 20 01:24:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:24:53 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 01:24:53 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 01:24:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:24:53 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 01:24:53 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 01:24:53 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 01:24:53 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 01:24:53 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 01:24:53 cheezum kernel:  [<c08837d1>] ? start_kernel+0x2dd/0x2e2
Mar 20 01:24:53 cheezum kernel:  [<c0883332>] ? kernel_init+0x141/0x141
Mar 20 01:24:53 cheezum kernel:  [<c088672b>] ? xen_start_kernel+0x4f7/0x4ff
Mar 20 01:25:01 cheezum /USR/SBIN/CRON[12816]: (root) CMD ([ -x /usr/lib/sysstat/sa1 ] && { [ -r "$DEFAULT" ] && . "$DEFAULT" ; [ "$ENABLED" = "true" ] && exec /usr/lib/sysstat/sa1 $SA1_OPTIONS 1 1 ; })
Mar 20 01:27:53 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=5280938 jiffies)
Mar 20 01:27:53 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 01:27:53 cheezum kernel: Call Trace:
Mar 20 01:27:53 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 01:27:53 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 01:27:53 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 01:27:53 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 01:27:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:27:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:27:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:27:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:27:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:27:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:27:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:27:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:27:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:27:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:27:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:27:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:27:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:27:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:27:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:27:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:27:53 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 01:27:53 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 01:27:53 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 01:27:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=5280870 jiffies)
Mar 20 01:27:53 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 01:27:53 cheezum kernel: Call Trace:
Mar 20 01:27:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 01:27:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 01:27:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:27:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:27:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:27:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:27:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:27:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:27:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:27:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:27:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:27:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:27:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:27:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:27:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:27:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:27:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:27:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:27:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 01:27:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 01:27:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:27:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 01:27:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 01:27:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 01:27:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 01:27:53 cheezum kernel:  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 01:27:53 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 01:27:53 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 01:27:53 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 01:27:53 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 01:27:53 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 01:27:53 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 01:27:53 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 01:27:53 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 01:27:53 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 01:27:53 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 01:30:54 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=5460971 jiffies)
Mar 20 01:30:54 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 01:30:54 cheezum kernel: Call Trace:
Mar 20 01:30:54 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 01:30:54 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 01:30:54 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 01:30:54 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 01:30:54 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:30:54 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:30:54 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:30:54 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:30:54 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:30:54 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:30:54 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:30:54 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:30:54 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:30:54 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:30:54 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:30:54 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:30:54 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:30:54 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:30:54 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:30:54 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:30:54 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 01:30:54 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 01:30:54 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 01:30:54 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=5460900 jiffies)
Mar 20 01:30:54 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 01:30:54 cheezum kernel: Call Trace:
Mar 20 01:30:54 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 01:30:54 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 01:30:54 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:30:54 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:30:54 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:30:54 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:30:54 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:30:54 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:30:54 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:30:54 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:30:54 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:30:54 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:30:54 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:30:54 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:30:54 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:30:54 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:30:54 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:30:54 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:30:54 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 01:30:54 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 01:30:54 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:30:54 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 01:30:54 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 01:30:54 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 01:30:54 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 01:30:54 cheezum kernel:  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 01:30:54 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 01:30:54 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 01:30:54 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 01:30:54 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 01:30:54 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 01:30:54 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 01:30:54 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 01:30:54 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 01:30:54 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 01:30:54 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 01:33:53 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=5641002 jiffies)
Mar 20 01:33:53 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 01:33:53 cheezum kernel: Call Trace:
Mar 20 01:33:53 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 01:33:53 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 01:33:53 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 01:33:53 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 01:33:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:33:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:33:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:33:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:33:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:33:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:33:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:33:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:33:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:33:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:33:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:33:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:33:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:33:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:33:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:33:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:33:53 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 01:33:53 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 01:33:53 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 01:33:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=5640930 jiffies)
Mar 20 01:33:53 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 01:33:53 cheezum kernel: Call Trace:
Mar 20 01:33:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 01:33:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 01:33:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:33:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:33:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:33:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:33:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:33:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:33:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:33:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:33:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:33:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:33:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:33:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:33:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:33:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:33:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:33:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:33:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 01:33:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 01:33:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:33:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 01:33:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 01:33:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 01:33:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 01:33:53 cheezum kernel:  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 01:33:53 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 01:33:53 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 01:33:53 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 01:33:53 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 01:33:53 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 01:33:53 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 01:33:53 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 01:33:53 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 01:33:53 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 01:33:53 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 01:35:02 cheezum /USR/SBIN/CRON[12835]: (root) CMD ([ -x /usr/lib/sysstat/sa1 ] && { [ -r "$DEFAULT" ] && . "$DEFAULT" ; [ "$ENABLED" = "true" ] && exec /usr/lib/sysstat/sa1 $SA1_OPTIONS 1 1 ; })
Mar 20 01:36:53 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=5821036 jiffies)
Mar 20 01:36:53 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 01:36:53 cheezum kernel: Call Trace:
Mar 20 01:36:53 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 01:36:53 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 01:36:53 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 01:36:53 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 01:36:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:36:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:36:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:36:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:36:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:36:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:36:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:36:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:36:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:36:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:36:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:36:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:36:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:36:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:36:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:36:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:36:53 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 01:36:53 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 01:36:53 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 01:36:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=5820960 jiffies)
Mar 20 01:36:53 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 01:36:53 cheezum kernel: Call Trace:
Mar 20 01:36:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 01:36:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 01:36:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:36:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:36:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:36:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:36:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:36:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:36:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:36:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:36:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:36:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:36:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:36:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:36:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:36:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:36:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:36:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:36:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 01:36:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 01:36:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:36:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 01:36:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 01:36:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 01:36:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 01:36:53 cheezum kernel:  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 01:36:53 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 01:36:53 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 01:36:53 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 01:36:53 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 01:36:53 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 01:36:53 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 01:36:53 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 01:36:53 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 01:36:53 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 01:36:53 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 01:39:53 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=6001068 jiffies)
Mar 20 01:39:53 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 01:39:53 cheezum kernel: Call Trace:
Mar 20 01:39:53 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 01:39:53 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 01:39:53 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 01:39:53 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 01:39:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:39:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:39:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:39:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:39:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:39:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:39:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:39:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:39:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:39:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:39:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:39:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:39:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:39:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:39:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:39:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:39:53 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 01:39:53 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 01:39:53 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 01:39:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=6000990 jiffies)
Mar 20 01:39:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 0 (t=6000990 jiffies)
Mar 20 01:39:53 cheezum kernel: Pid: 0, comm: swapper/0 Not tainted 3.2.1-linode40 #1
Mar 20 01:39:53 cheezum kernel: Call Trace:
Mar 20 01:39:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 01:39:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 01:39:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:39:53 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 01:39:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:39:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:39:53 cheezum kernel: Call Trace:
Mar 20 01:39:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 01:39:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 01:39:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:39:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:39:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:39:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:39:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:39:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:39:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:39:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:39:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:39:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:39:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:39:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:39:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:39:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:39:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:39:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:39:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:39:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:39:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:39:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:39:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:39:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:39:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:39:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 01:39:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 01:39:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:39:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:39:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 01:39:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:39:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 01:39:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 01:39:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 01:39:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:39:53 cheezum kernel:  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 01:39:53 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 01:39:53 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 01:39:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:39:53 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 01:39:53 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 01:39:53 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 01:39:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:39:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:39:53 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 01:39:53 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 01:39:53 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 01:39:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 01:39:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 01:39:53 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 01:39:53 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 01:39:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:39:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 01:39:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 01:39:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 01:39:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 01:39:53 cheezum kernel:  [<c05de8ba>] ? rt_garbage_collect+0x12a/0x400
Mar 20 01:39:53 cheezum kernel:  [<c0564add>] ? dst_alloc+0xfd/0x140
Mar 20 01:39:53 cheezum kernel:  [<c05dfc97>] ? rt_dst_alloc+0x37/0x40
Mar 20 01:39:53 cheezum kernel:  [<c05e16cf>] ? ip_route_input_slow+0x28f/0x560
Mar 20 01:39:53 cheezum kernel:  [<c05e0010>] ? __mkroute_output+0x10/0x2e0
Mar 20 01:39:53 cheezum kernel:  [<c059a2e4>] ? nf_hook_slow+0x64/0x130
Mar 20 01:39:53 cheezum kernel:  [<c05e3213>] ? ip_rcv_finish+0x243/0x380
Mar 20 01:39:53 cheezum kernel:  [<c055bad8>] ? __netif_receive_skb+0x2e8/0x350
Mar 20 01:39:53 cheezum kernel:  [<c055d3f7>] ? netif_receive_skb+0x67/0x70
Mar 20 01:39:53 cheezum kernel:  [<c04fcdb1>] ? handle_incoming_queue+0x1a1/0x270
Mar 20 01:39:53 cheezum kernel:  [<c04fd0a4>] ? xennet_poll+0x224/0x570
Mar 20 01:39:53 cheezum kernel:  [<c055db8a>] ? net_rx_action+0xea/0x190
Mar 20 01:39:53 cheezum kernel:  [<c013ec5c>] ? __do_softirq+0x7c/0x110
Mar 20 01:39:53 cheezum kernel:  [<c013ebe0>] ? irq_enter+0x60/0x60
Mar 20 01:39:53 cheezum kernel:  <IRQ>  [<c013ea3e>] ? irq_exit+0x6e/0xa0
Mar 20 01:39:53 cheezum kernel:  [<c048d9dd>] ? xen_evtchn_do_upcall+0x1d/0x30
Mar 20 01:39:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:39:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:39:53 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 01:39:53 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 01:39:53 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 01:39:53 cheezum kernel:  [<c08837d1>] ? start_kernel+0x2dd/0x2e2
Mar 20 01:39:53 cheezum kernel:  [<c0883332>] ? kernel_init+0x141/0x141
Mar 20 01:39:53 cheezum kernel:  [<c088672b>] ? xen_start_kernel+0x4f7/0x4ff
Mar 20 01:42:56 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=6181100 jiffies)
Mar 20 01:42:57 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 01:42:57 cheezum kernel: Call Trace:
Mar 20 01:42:57 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 01:42:57 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 01:42:57 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 01:42:57 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 01:42:57 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:42:57 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:42:57 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:42:57 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:42:57 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:42:57 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:42:57 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:42:57 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:42:57 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:42:57 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:42:57 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:42:57 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:42:57 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:42:57 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:42:57 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:42:57 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:42:57 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 01:42:57 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 01:42:57 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 01:42:57 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=6181020 jiffies)
Mar 20 01:42:57 cheezum kernel: INFO: rcu_sched detected stall on CPU 0 (t=6181020 jiffies)
Mar 20 01:42:57 cheezum kernel: Pid: 0, comm: swapper/0 Not tainted 3.2.1-linode40 #1
Mar 20 01:42:57 cheezum kernel: Call Trace:
Mar 20 01:42:57 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 01:42:57 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 01:42:57 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:42:57 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:42:57 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 01:42:57 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:42:57 cheezum kernel: Call Trace:
Mar 20 01:42:57 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:42:57 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:42:57 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 01:42:57 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 01:42:57 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:42:57 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:42:57 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:42:57 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:42:57 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:42:57 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:42:57 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:42:57 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:42:57 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:42:57 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:42:57 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:42:57 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:42:57 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:42:57 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:42:57 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:42:57 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:42:57 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 01:42:57 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 01:42:57 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:42:57 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:42:57 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 01:42:57 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:42:57 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 01:42:57 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 01:42:57 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:42:57 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 01:42:57 cheezum kernel:  [<c05de8ba>] ? rt_garbage_collect+0x12a/0x400
Mar 20 01:42:57 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:42:57 cheezum kernel:  [<c0564add>] ? dst_alloc+0xfd/0x140
Mar 20 01:42:57 cheezum kernel:  [<c05dfc97>] ? rt_dst_alloc+0x37/0x40
Mar 20 01:42:57 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:42:57 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:42:57 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:42:57 cheezum kernel:  [<c05e16cf>] ? ip_route_input_slow+0x28f/0x560
Mar 20 01:42:57 cheezum kernel:  [<c05e0010>] ? __mkroute_output+0x10/0x2e0
Mar 20 01:42:57 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:42:57 cheezum kernel:  [<c059a2e4>] ? nf_hook_slow+0x64/0x130
Mar 20 01:42:57 cheezum kernel:  [<c05e3213>] ? ip_rcv_finish+0x243/0x380
Mar 20 01:42:57 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:42:57 cheezum kernel:  [<c055bad8>] ? __netif_receive_skb+0x2e8/0x350
Mar 20 01:42:57 cheezum kernel:  [<c055d3f7>] ? netif_receive_skb+0x67/0x70
Mar 20 01:42:57 cheezum kernel:  [<c04fcdb1>] ? handle_incoming_queue+0x1a1/0x270
Mar 20 01:42:57 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:42:57 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 01:42:57 cheezum kernel:  [<c04fd0a4>] ? xennet_poll+0x224/0x570
Mar 20 01:42:57 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 01:42:57 cheezum kernel:  [<c055db8a>] ? net_rx_action+0xea/0x190
Mar 20 01:42:57 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:42:57 cheezum kernel:  [<c013ec5c>] ? __do_softirq+0x7c/0x110
Mar 20 01:42:57 cheezum kernel:  [<c013ebe0>] ? irq_enter+0x60/0x60
Mar 20 01:42:57 cheezum kernel:  <IRQ>  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 01:42:57 cheezum kernel:  [<c013ea3e>] ? irq_exit+0x6e/0xa0
Mar 20 01:42:57 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 01:42:57 cheezum kernel:  [<c048d9dd>] ? xen_evtchn_do_upcall+0x1d/0x30
Mar 20 01:42:57 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:42:57 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:42:57 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 01:42:57 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 01:42:57 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 01:42:57 cheezum kernel:  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 01:42:57 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 01:42:57 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 01:42:57 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 01:42:57 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 01:42:57 cheezum kernel:  [<c08837d1>] ? start_kernel+0x2dd/0x2e2
Mar 20 01:42:57 cheezum kernel:  [<c0883332>] ? kernel_init+0x141/0x141
Mar 20 01:42:57 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 01:42:57 cheezum kernel:  [<c088672b>] ? xen_start_kernel+0x4f7/0x4ff
Mar 20 01:42:57 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 01:42:57 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 01:42:57 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 01:42:57 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 01:42:57 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 01:42:57 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 01:42:57 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 01:45:02 cheezum /USR/SBIN/CRON[12853]: (root) CMD ([ -x /usr/lib/sysstat/sa1 ] && { [ -r "$DEFAULT" ] && . "$DEFAULT" ; [ "$ENABLED" = "true" ] && exec /usr/lib/sysstat/sa1 $SA1_OPTIONS 1 1 ; })
Mar 20 01:45:53 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=6361133 jiffies)
Mar 20 01:45:53 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 01:45:53 cheezum kernel: Call Trace:
Mar 20 01:45:53 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 01:45:53 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 01:45:53 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 01:45:53 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 01:45:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:45:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:45:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:45:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:45:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:45:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:45:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:45:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:45:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:45:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:45:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:45:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:45:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:45:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:45:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:45:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:45:53 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 01:45:53 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 01:45:53 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 01:45:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=6361050 jiffies)
Mar 20 01:45:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 0 (t=6361050 jiffies)
Mar 20 01:45:53 cheezum kernel: Pid: 0, comm: swapper/0 Not tainted 3.2.1-linode40 #1
Mar 20 01:45:53 cheezum kernel: Call Trace:
Mar 20 01:45:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 01:45:53 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 01:45:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 01:45:53 cheezum kernel: Call Trace:
Mar 20 01:45:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 01:45:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:45:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 01:45:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:45:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:45:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:45:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:45:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:45:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:45:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:45:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:45:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:45:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:45:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:45:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:45:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:45:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:45:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:45:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:45:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:45:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:45:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:45:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:45:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:45:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:45:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:45:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:45:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:45:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:45:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:45:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:45:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:45:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 01:45:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 01:45:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:45:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:45:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 01:45:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 01:45:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 01:45:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 01:45:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 01:45:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 01:45:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:45:53 cheezum kernel:  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 01:45:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 01:45:53 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 01:45:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 01:45:53 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 01:45:53 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 01:45:53 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 01:45:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 01:45:53 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 01:45:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 01:45:53 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 01:45:53 cheezum kernel:  [<c05de8ba>] ? rt_garbage_collect+0x12a/0x400
Mar 20 01:45:53 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 01:45:53 cheezum kernel:  [<c0564add>] ? dst_alloc+0xfd/0x140
Mar 20 01:45:53 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 01:45:53 cheezum kernel:  [<c05dfc97>] ? rt_dst_alloc+0x37/0x40
Mar 20 01:45:53 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 01:45:53 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 01:45:53 cheezum kernel:  [<c05e16cf>] ? ip_route_input_slow+0x28f/0x560
Mar 20 01:45:53 cheezum kernel:  [<c05e0010>] ? __mkroute_output+0x10/0x2e0
Mar 20 01:45:53 cheezum kernel:  [<c059a2e4>] ? nf_hook_slow+0x64/0x130
Mar 20 01:45:53 cheezum kernel:  [<c05e3213>] ? ip_rcv_finish+0x243/0x380
Mar 20 01:45:53 cheezum kernel:  [<c055bad8>] ? __netif_receive_skb+0x2e8/0x350
Mar 20 01:45:53 cheezum kernel:  [<c055d3f7>] ? netif_receive_skb+0x67/0x70
Mar 20 01:45:53 cheezum kernel:  [<c04fcdb1>] ? handle_incoming_queue+0x1a1/0x270
Mar 20 01:45:53 cheezum kernel:  [<c04fd0a4>] ? xennet_poll+0x224/0x570
Mar 20 01:45:53 cheezum kernel:  [<c055db8a>] ? net_rx_action+0xea/0x190
Mar 20 01:45:53 cheezum kernel:  [<c013ec5c>] ? __do_softirq+0x7c/0x110
Mar 20 01:45:53 cheezum kernel:  [<c013ebe0>] ? irq_enter+0x60/0x60
Mar 20 01:45:53 cheezum kernel:  <IRQ>  [<c013ea3e>] ? irq_exit+0x6e/0xa0
Mar 20 01:45:53 cheezum kernel:  [<c048d9dd>] ? xen_evtchn_do_upcall+0x1d/0x30
Mar 20 01:45:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:45:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:45:53 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 01:45:53 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 01:45:53 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 01:45:53 cheezum kernel:  [<c08837d1>] ? start_kernel+0x2dd/0x2e2
Mar 20 01:45:53 cheezum kernel:  [<c0883332>] ? kernel_init+0x141/0x141
Mar 20 01:45:53 cheezum kernel:  [<c088672b>] ? xen_start_kernel+0x4f7/0x4ff
Mar 20 01:48:53 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=6541165 jiffies)
Mar 20 01:48:53 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 01:48:53 cheezum kernel: Call Trace:
Mar 20 01:48:53 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 01:48:53 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 01:48:53 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 01:48:53 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 01:48:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:48:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:48:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:48:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:48:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:48:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:48:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:48:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:48:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:48:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:48:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:48:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:48:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:48:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:48:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:48:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:48:53 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 01:48:53 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 01:48:53 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 01:48:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=6541080 jiffies)
Mar 20 01:48:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 0 (t=6541080 jiffies)
Mar 20 01:48:53 cheezum kernel: Pid: 0, comm: swapper/0 Not tainted 3.2.1-linode40 #1
Mar 20 01:48:53 cheezum kernel: Call Trace:
Mar 20 01:48:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 01:48:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 01:48:53 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 01:48:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:48:53 cheezum kernel: Call Trace:
Mar 20 01:48:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:48:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:48:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 01:48:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 01:48:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:48:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:48:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:48:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:48:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:48:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:48:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:48:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:48:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:48:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:48:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:48:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:48:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:48:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:48:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:48:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:48:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:48:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:48:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:48:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:48:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 01:48:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:48:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 01:48:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:48:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:48:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 01:48:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 01:48:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:48:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 01:48:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:48:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 01:48:53 cheezum kernel:  [<c05de8ba>] ? rt_garbage_collect+0x12a/0x400
Mar 20 01:48:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:48:53 cheezum kernel:  [<c0564add>] ? dst_alloc+0xfd/0x140
Mar 20 01:48:53 cheezum kernel:  [<c05dfc97>] ? rt_dst_alloc+0x37/0x40
Mar 20 01:48:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:48:53 cheezum kernel:  [<c05e16cf>] ? ip_route_input_slow+0x28f/0x560
Mar 20 01:48:53 cheezum kernel:  [<c05e0010>] ? __mkroute_output+0x10/0x2e0
Mar 20 01:48:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:48:53 cheezum kernel:  [<c059a2e4>] ? nf_hook_slow+0x64/0x130
Mar 20 01:48:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:48:53 cheezum kernel:  [<c05e3213>] ? ip_rcv_finish+0x243/0x380
Mar 20 01:48:53 cheezum kernel:  [<c055bad8>] ? __netif_receive_skb+0x2e8/0x350
Mar 20 01:48:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 01:48:53 cheezum kernel:  [<c055d3f7>] ? netif_receive_skb+0x67/0x70
Mar 20 01:48:53 cheezum kernel:  [<c04fcdb1>] ? handle_incoming_queue+0x1a1/0x270
Mar 20 01:48:53 cheezum kernel:  [<c04fd0a4>] ? xennet_poll+0x224/0x570
Mar 20 01:48:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 01:48:53 cheezum kernel:  [<c055db8a>] ? net_rx_action+0xea/0x190
Mar 20 01:48:53 cheezum kernel:  [<c013ec5c>] ? __do_softirq+0x7c/0x110
Mar 20 01:48:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:48:53 cheezum kernel:  [<c013ebe0>] ? irq_enter+0x60/0x60
Mar 20 01:48:53 cheezum kernel:  <IRQ>  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 01:48:53 cheezum kernel:  [<c013ea3e>] ? irq_exit+0x6e/0xa0
Mar 20 01:48:53 cheezum kernel:  [<c048d9dd>] ? xen_evtchn_do_upcall+0x1d/0x30
Mar 20 01:48:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 01:48:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:48:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 01:48:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:48:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 01:48:53 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 01:48:53 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 01:48:53 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 01:48:53 cheezum kernel:  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 01:48:53 cheezum kernel:  [<c08837d1>] ? start_kernel+0x2dd/0x2e2
Mar 20 01:48:53 cheezum kernel:  [<c0883332>] ? kernel_init+0x141/0x141
Mar 20 01:48:53 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 01:48:53 cheezum kernel:  [<c088672b>] ? xen_start_kernel+0x4f7/0x4ff
Mar 20 01:48:53 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 01:48:53 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 01:48:53 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 01:48:53 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 01:48:53 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 01:48:53 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 01:48:53 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 01:48:53 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 01:48:53 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 01:51:53 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=6721197 jiffies)
Mar 20 01:51:53 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 01:51:53 cheezum kernel: Call Trace:
Mar 20 01:51:53 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 01:51:53 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 01:51:53 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 01:51:53 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 01:51:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:51:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:51:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:51:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:51:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:51:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:51:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:51:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:51:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:51:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:51:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:51:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:51:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:51:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:51:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:51:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:51:53 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 01:51:53 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 01:51:53 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 01:51:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=6721110 jiffies)
Mar 20 01:51:53 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 01:51:53 cheezum kernel: Call Trace:
Mar 20 01:51:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 01:51:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 01:51:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:51:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:51:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:51:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:51:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:51:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:51:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:51:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:51:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:51:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:51:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:51:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:51:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:51:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:51:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:51:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:51:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 01:51:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 01:51:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:51:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 01:51:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 01:51:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 01:51:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 01:51:53 cheezum kernel:  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 01:51:53 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 01:51:53 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 01:51:53 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 01:51:53 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 01:51:53 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 01:51:53 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 01:51:53 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 01:51:53 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 01:51:53 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 01:51:53 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 01:54:53 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=6901228 jiffies)
Mar 20 01:54:53 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 01:54:53 cheezum kernel: Call Trace:
Mar 20 01:54:53 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 01:54:53 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 01:54:53 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 01:54:53 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 01:54:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:54:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:54:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:54:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:54:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:54:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:54:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:54:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:54:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:54:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:54:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:54:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:54:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:54:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:54:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:54:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:54:53 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 01:54:53 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 01:54:53 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 01:54:53 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=6901140 jiffies)
Mar 20 01:54:53 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 01:54:53 cheezum kernel: Call Trace:
Mar 20 01:54:53 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 01:54:53 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 01:54:53 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:54:53 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:54:53 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:54:53 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:54:53 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:54:53 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:54:53 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:54:53 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:54:53 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:54:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:54:53 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:54:53 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:54:53 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:54:53 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:54:53 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:54:53 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:54:53 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 01:54:53 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 01:54:53 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:54:53 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 01:54:53 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 01:54:53 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 01:54:53 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 01:54:53 cheezum kernel:  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 01:54:53 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 01:54:53 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 01:54:53 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 01:54:53 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 01:54:53 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 01:54:53 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 01:54:53 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 01:54:53 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 01:54:53 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 01:54:53 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 01:57:54 cheezum kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 0} (detected by 2, t=7081261 jiffies)
Mar 20 01:57:54 cheezum kernel: Pid: 0, comm: swapper/2 Not tainted 3.2.1-linode40 #1
Mar 20 01:57:54 cheezum kernel: Call Trace:
Mar 20 01:57:54 cheezum kernel:  [<c0186598>] ? print_other_cpu_stall+0x148/0x1a0
Mar 20 01:57:54 cheezum kernel:  [<c018664a>] ? __rcu_pending+0x5a/0x210
Mar 20 01:57:54 cheezum kernel:  [<c0106804>] ? check_events+0x8/0xc
Mar 20 01:57:54 cheezum kernel:  [<c01868ee>] ? rcu_check_callbacks+0xee/0x100
Mar 20 01:57:54 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:57:54 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:57:54 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:57:54 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:57:54 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:57:54 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:57:54 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:57:54 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:57:54 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:57:54 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:57:54 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:57:54 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:57:54 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:57:54 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:57:54 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:57:54 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:57:54 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 01:57:54 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 01:57:54 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 01:57:54 cheezum kernel: INFO: rcu_sched detected stall on CPU 0 (t=7081170 jiffies)
Mar 20 01:57:54 cheezum kernel: INFO: rcu_sched detected stall on CPU 1 (t=7081170 jiffies)
Mar 20 01:57:54 cheezum kernel: Pid: 8, comm: kworker/1:0 Not tainted 3.2.1-linode40 #1
Mar 20 01:57:54 cheezum kernel: Call Trace:
Mar 20 01:57:54 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 01:57:54 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 01:57:54 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:57:54 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:57:54 cheezum kernel: Pid: 0, comm: swapper/0 Not tainted 3.2.1-linode40 #1
Mar 20 01:57:54 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:57:54 cheezum kernel: Call Trace:
Mar 20 01:57:54 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:57:54 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:57:54 cheezum kernel:  [<c01866e1>] ? __rcu_pending+0xf1/0x210
Mar 20 01:57:54 cheezum kernel:  [<c0186846>] ? rcu_check_callbacks+0x46/0x100
Mar 20 01:57:54 cheezum kernel:  [<c01453a5>] ? update_process_times+0x35/0x70
Mar 20 01:57:54 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:57:54 cheezum kernel:  [<c0163189>] ? tick_sched_timer+0x59/0xb0
Mar 20 01:57:54 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:57:54 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:57:54 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:57:54 cheezum kernel:  [<c0156865>] ? __remove_hrtimer+0x45/0xa0
Mar 20 01:57:54 cheezum kernel:  [<c0163130>] ? tick_nohz_handler+0xe0/0xe0
Mar 20 01:57:54 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:57:54 cheezum kernel:  [<c01569ed>] ? __run_hrtimer+0x4d/0xf0
Mar 20 01:57:54 cheezum kernel:  [<c0156d11>] ? hrtimer_interrupt+0x101/0x2c0
Mar 20 01:57:54 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:57:54 cheezum kernel:  [<c01066a7>] ? xen_timer_interrupt+0x17/0x30
Mar 20 01:57:54 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:57:54 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:57:54 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:57:54 cheezum kernel:  [<c0180fd7>] ? handle_irq_event_percpu+0x37/0x140
Mar 20 01:57:54 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:57:54 cheezum kernel:  [<c01833b5>] ? irq_get_irq_data+0x5/0x10
Mar 20 01:57:54 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:57:54 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 01:57:54 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:57:54 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 01:57:54 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:57:54 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 01:57:54 cheezum kernel:  [<c048cf80>] ? evtchn_from_irq+0x10/0x40
Mar 20 01:57:54 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 01:57:54 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 01:57:54 cheezum kernel:  [<c048c885>] ? info_for_irq+0x5/0x20
Mar 20 01:57:54 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 01:57:54 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:57:54 cheezum kernel:  [<c05df866>] ? rt_check_expire+0x106/0x310
Mar 20 01:57:54 cheezum kernel:  [<c05dfa75>] ? rt_worker_func+0x5/0x20
Mar 20 01:57:54 cheezum kernel:  [<c014f158>] ? process_one_work+0xf8/0x340
Mar 20 01:57:54 cheezum kernel:  [<c048c742>] ? __xen_evtchn_do_upcall+0x162/0x210
Mar 20 01:57:54 cheezum kernel:  [<c0101227>] ? hypercall_page+0x227/0x1000
Mar 20 01:57:54 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 01:57:54 cheezum kernel:  [<c048d9d8>] ? xen_evtchn_do_upcall+0x18/0x30
Mar 20 01:57:54 cheezum kernel:  [<c05dfa70>] ? rt_check_expire+0x310/0x310
Mar 20 01:57:54 cheezum kernel:  [<c014f740>] ? worker_thread+0x120/0x3c0
Mar 20 01:57:54 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:57:54 cheezum kernel:  [<c014f620>] ? rescuer_thread+0x280/0x280
Mar 20 01:57:54 cheezum kernel:  [<c018007b>] ? audit_tag_tree+0x23b/0x380
Mar 20 01:57:54 cheezum kernel:  [<c0152c94>] ? kthread+0x74/0x80
Mar 20 01:57:54 cheezum kernel:  [<c0152c20>] ? kthread_worker_fn+0x170/0x170
Mar 20 01:57:54 cheezum kernel:  [<c04800d8>] ? build_tree+0x428/0x550
Mar 20 01:57:54 cheezum kernel:  [<c06e2436>] ? kernel_thread_helper+0x6/0x10
Mar 20 01:57:54 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:57:54 cheezum kernel:  [<c048cff7>] ? xen_poll_irq_timeout+0x47/0x60
Mar 20 01:57:54 cheezum kernel:  [<c0107c05>] ? xen_spin_lock_slow+0x65/0xe0
Mar 20 01:57:54 cheezum kernel:  [<c0107d37>] ? xen_spin_lock+0x47/0x50
Mar 20 01:57:54 cheezum kernel:  [<c06e15df>] ? _raw_spin_lock_bh+0xf/0x20
Mar 20 01:57:54 cheezum kernel:  [<c05de8ba>] ? rt_garbage_collect+0x12a/0x400
Mar 20 01:57:54 cheezum kernel:  [<c0564add>] ? dst_alloc+0xfd/0x140
Mar 20 01:57:54 cheezum kernel:  [<c05dfc97>] ? rt_dst_alloc+0x37/0x40
Mar 20 01:57:54 cheezum kernel:  [<c05e16cf>] ? ip_route_input_slow+0x28f/0x560
Mar 20 01:57:54 cheezum kernel:  [<c05e0010>] ? __mkroute_output+0x10/0x2e0
Mar 20 01:57:54 cheezum kernel:  [<c059a2e4>] ? nf_hook_slow+0x64/0x130
Mar 20 01:57:54 cheezum kernel:  [<c05e3213>] ? ip_rcv_finish+0x243/0x380
Mar 20 01:57:54 cheezum kernel:  [<c055bad8>] ? __netif_receive_skb+0x2e8/0x350
Mar 20 01:57:54 cheezum kernel:  [<c055d3f7>] ? netif_receive_skb+0x67/0x70
Mar 20 01:57:54 cheezum kernel:  [<c04fcdb1>] ? handle_incoming_queue+0x1a1/0x270
Mar 20 01:57:54 cheezum kernel:  [<c04fd0a4>] ? xennet_poll+0x224/0x570
Mar 20 01:57:54 cheezum kernel:  [<c055db8a>] ? net_rx_action+0xea/0x190
Mar 20 01:57:54 cheezum kernel:  [<c013ec5c>] ? __do_softirq+0x7c/0x110
Mar 20 01:57:54 cheezum kernel:  [<c013ebe0>] ? irq_enter+0x60/0x60
Mar 20 01:57:54 cheezum kernel:  <IRQ>  [<c013ea3e>] ? irq_exit+0x6e/0xa0
Mar 20 01:57:54 cheezum kernel:  [<c048d9dd>] ? xen_evtchn_do_upcall+0x1d/0x30
Mar 20 01:57:54 cheezum kernel:  [<c06e2487>] ? xen_do_upcall+0x7/0xc
Mar 20 01:57:54 cheezum kernel:  [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Mar 20 01:57:54 cheezum kernel:  [<c010609f>] ? xen_safe_halt+0xf/0x20
Mar 20 01:57:54 cheezum kernel:  [<c010fa5f>] ? default_idle+0x2f/0x60
Mar 20 01:57:54 cheezum kernel:  [<c0108457>] ? cpu_idle+0x47/0x80
Mar 20 01:57:54 cheezum kernel:  [<c08837d1>] ? start_kernel+0x2dd/0x2e2
Mar 20 01:57:54 cheezum kernel:  [<c0883332>] ? kernel_init+0x141/0x141
Mar 20 01:57:54 cheezum kernel:  [<c088672b>] ? xen_start_kernel+0x4f7/0x4ff
Mar 20 01:58:37 cheezum kernel: <     rss cpu oom_adj oom_score_adj name
Mar 20 01:58:38 cheezum kernel: [ 1121]     0  1121      569        0   1     -17         -1000 udevd
Mar 20 01:58:38 cheezum kernel: [ 2613]     0  2613      656        0   1       0             0 login
Mar 20 01:58:38 cheezum kernel: [ 2650]   106  2650      497       14   2       0             0 syslogd
Mar 20 01:58:38 cheezum kernel: [ 2671]     0  2671      481        4   2       0             0 dd
Mar 20 01:58:38 cheezum kernel: [ 2673]   105  2673      631        7   2       0             0 klogd
Mar 20 01:58:38 cheezum kernel: [ 2691]   101  2691      872        5   2       0             0 dnsmasq
Mar 20 01:58:38 cheezum kernel: [ 2712]     0  2712     1342        6   1     -17         -1000 sshd
Mar 20 01:58:38 cheezum kernel: [ 2771]  1000  2771     3085      431   2     -17         -1000 screen
Mar 20 01:58:38 cheezum kernel: [ 2788]  1000  2788     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2804]  1000  2804     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2820]  1000  2820     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2836]  1000  2836     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2858]  1000  2858     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2887]  1000  2887     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2905]  1000  2905     1451        0   3     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2965]  1000  2965     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2987]  1000  2987     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3026]  1000  3026     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3054]  1000  3054     1451        0   2     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3070]   110  3070    10442      226   2       0             0 ntpd
Mar 20 01:58:38 cheezum kernel: [ 3109]     0  3109      539        8   2       0             0 cron
Mar 20 01:58:38 cheezum kernel: [ 3146]     1  3146      441        9   2       0             0 uptimed
Mar 20 01:58:38 cheezum kernel: [ 3189]  1000  3189    13778      782   3     -17         -1000 irssi
Mar 20 01:58:38 cheezum kernel: [ 3203]  1001  3203     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3238]  1000  3238     1450        0   1       0             0 bash
Mar 20 01:58:38 cheezum kernel: [ 3453]  1000  3453     1451        0   3     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3652]     0  3652      786        7   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 3656]     0  3656      786        8   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 3658]     0  3658      786        6   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 3696]     0  3696     1816      101   3     -17         -1000 tcpdump
Mar 20 01:58:38 cheezum kernel: [12409]  1001 12409     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [12430]  1001 12430     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15137]  1000 15137     1451        0   2     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15154]  1000 15154      823        0   0     -17         -1000 less
Mar 20 01:58:38 cheezum kernel: [15589]  1000 15589     1451        0   3     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15606]  1000 15606      636       48   3     -17         -1000 htop
Mar 20 01:58:38 cheezum kernel: [15607]  1000 15607     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15623]  1000 15623     1219        0   0     -17         -1000 python
Mar 20 01:58:38 cheezum kernel: [15710]  1000 15710     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15726]     0 15726     2046      315   2     -17         -1000 iotop
Mar 20 01:58:38 cheezum kernel: [15727]  1000 15727     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15743]  1000 15743      473       12   3     -17         -1000 vmstat
Mar 20 01:58:38 cheezum kernel: [15760]  1000 15760     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15776]  1000 15776      516       19   2     -17         -1000 iostat
Mar 20 01:58:38 cheezum kernel: [15825]  1000 15825     1451        0   2     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15841]     0 15841     2619       37   2     -17         -1000 tcpdump
Mar 20 01:58:38 cheezum kernel: [15867]  1000 15867     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15883]  1000 15883     4229        3   0     -17         -1000 nano
Mar 20 01:58:38 cheezum kernel: [16026]  1000 16026     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [16050]  1000 16050     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [16214]     0 16214      786        9   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [16660]     0 16660      786        7   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 7028]  1000  7028     1451        1   3     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [14552]  1000 14552     1451        0   2     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [16812]   111 16812     2713        0   1     -17         -1000 nsd
Mar 20 01:58:38 cheezum kernel: [19295]  1000 19295      823        0   3     -17         -1000 less
Mar 20 01:58:38 cheezum kernel: < pages non-shared
Mar 20 01:58:38 cheezum kernel: [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Mar 20 01:58:38 cheezum kernel: [ 1121]     0  1121      569        0   1     -17         -1000 udevd
Mar 20 01:58:38 cheezum kernel: [ 2613]     0  2613      656        0   1       0             0 login
Mar 20 01:58:38 cheezum kernel: [ 2650]   106  2650      497       14   2       0             0 syslogd
Mar 20 01:58:38 cheezum kernel: [ 2671]     0  2671      481        4   0       0             0 dd
Mar 20 01:58:38 cheezum kernel: [ 2673]   105  2673      631        7   2       0             0 klogd
Mar 20 01:58:38 cheezum kernel: [ 2691]   101  2691      872        5   2       0             0 dnsmasq
Mar 20 01:58:38 cheezum kernel: [ 2712]     0  2712     1342        6   1     -17         -1000 sshd
Mar 20 01:58:38 cheezum kernel: [ 2771]  1000  2771     3085      431   2     -17         -1000 screen
Mar 20 01:58:38 cheezum kernel: [ 2788]  1000  2788     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2804]  1000  2804     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2820]  1000  2820     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2836]  1000  2836     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2858]  1000  2858     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2887]  1000  2887     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2905]  1000  2905     1451        0   3     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2965]  1000  2965     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2987]  1000  2987     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3026]  1000  3026     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3054]  1000  3054     1451        0   2     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3109]     0  3109      539        8   3       0             0 cron
Mar 20 01:58:38 cheezum kernel: [ 3146]     1  3146      441        9   2       0             0 uptimed
Mar 20 01:58:38 cheezum kernel: [ 3189]  1000  3189    13778      782   3     -17         -1000 irssi
Mar 20 01:58:38 cheezum kernel: [ 3203]  1001  3203     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3238]  1000  3238     1450        0   1       0             0 bash
Mar 20 01:58:38 cheezum kernel: [ 3453]  1000  3453     1451        0   3     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3652]     0  3652      786        7   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 3656]     0  3656      786        8   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 3658]     0  3658      786        6   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 3696]     0  3696     1816      101   3     -17         -1000 tcpdump
Mar 20 01:58:38 cheezum kernel: [12409]  1001 12409     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [12430]  1001 12430     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15137]  1000 15137     1451        0   2     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15154]  1000 15154      823        0   0     -17         -1000 less
Mar 20 01:58:38 cheezum kernel: [15589]  1000 15589     1451        0   3     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15606]  1000 15606      636       48   3     -17         -1000 htop
Mar 20 01:58:38 cheezum kernel: [15607]  1000 15607     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15623]  1000 15623     1219        0   0     -17         -1000 python
Mar 20 01:58:38 cheezum kernel: [15710]  1000 15710     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15726]     0 15726     2046      315   2     -17         -1000 iotop
Mar 20 01:58:38 cheezum kernel: [15727]  1000 15727     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15743]  1000 15743      473       12   3     -17         -1000 vmstat
Mar 20 01:58:38 cheezum kernel: [15760]  1000 15760     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15776]  1000 15776      516       19   2     -17         -1000 iostat
Mar 20 01:58:38 cheezum kernel: [15825]  1000 15825     1451        0   2     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15841]     0 15841     2619       37   2     -17         -1000 tcpdump
Mar 20 01:58:38 cheezum kernel: [15867]  1000 15867     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15883]  1000 15883     4229        3   0     -17         -1000 nano
Mar 20 01:58:38 cheezum kernel: [16026]  1000 16026     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [16050]  1000 16050     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [16214]     0 16214      786        9   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [16660]     0 16660      786        7   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 7028]  1000  7028     1451        1   3     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [14552]  1000 14552     1451        0   2     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [16812]   111 16812     2713        0   1     -17         -1000 nsd
Mar 20 01:58:38 cheezum kernel: [19295]  1000 19295      823        0   3     -17         -1000 less
Mar 20 01:58:38 cheezum kernel: [32143]  1000 32143     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [32190]  1000 32190      823        0   1     -17         -1000 less
Mar 20 01:58:38 cheezum kernel: [ 4050]  1000  4050      786        8   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 1839]  1000  1839     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 1873]  1000  1873      823        0   1     -17         -1000 less
Mar 20 01:58:38 cheezum kernel: [31259]   112 31259    34250        0   1     -17         -1000 apt-cacher-ng
Mar 20 01:58:38 cheezum kernel: [13773]   111 13773     1048        0   3     -17         -1000 nsd
Mar 20 01:58:38 cheezum kernel: [13774]   111 13774     1192        0   2     -17         -1000 nsd
Mar 20 01:58:38 cheezum kernel: [31147]  1000 31147      474       13   2       0             0 vmstat
Mar 20 01:58:38 cheezum kernel: [  929]  1000   929     1452        0   2     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [  949]  1000   949      824        0   2     -17         -1000 less
Mar 20 01:58:38 cheezum kernel: [ 1021]    33  1021     1826      107   2       0             0 lighttpd
Mar 20 01:58:38 cheezum kernel: [ 1023]    33  1023    11996       52   3       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1024]    33  1024    11996       50   2       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1025]    33  1025    11996       43   2       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1026]    33  1026    11996       54   3       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1027]    33  1027     1619       69   2       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1031]    33  1031     1619       62   3       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1036]    33  1036     1619       68   3       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1038]    33  1038     1865       79   3       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1068]    33  1068     1586       65   2       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1089]    33  1089     1586       58   2       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1109]    33  1109     1586       61   2       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1135]    33  1135     1586       68   2       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 3980]  1000  3980     1452        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3998]  1000  3998      972        0   0     -17         -1000 nano
Mar 20 01:58:38 cheezum kernel: [ 4637]  1000  4637     1452        0   3     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 4654]  1000  4654      972        0   0     -17         -1000 nano
Mar 20 01:58:38 cheezum kernel: Out of memory: Kill process 1021 (lighttpd) score 2 or sacrifice child
Mar 20 01:58:38 cheezum kernel: Killed process 1023 (python) total-vm:47984kB, anon-rss:208kB, file-rss:0kB
Mar 20 01:58:38 cheezum kernel: cron invoked oom-killer: gfp_mask=0xd0, order=1, oom_adj=0, oom_score_adj=0
Mar 20 01:58:38 cheezum kernel: cron cpuset=/ mems_allowed=0
Mar 20 01:58:38 cheezum kernel: Pid: 3109, comm: cron Not tainted 3.2.1-linode40 #1
Mar 20 01:58:38 cheezum kernel: Call Trace:
Mar 20 01:58:38 cheezum kernel:  [<c01958fd>] ? T.662+0x7d/0x1b0
Mar 20 01:58:38 cheezum kernel:  [<c01067fb>] ? xen_restore_fl_direct_reloc+0x4/0x4
Mar 20 01:58:38 cheezum kernel:  [<c06e1431>] ? _raw_spin_unlock_irqrestore+0x11/0x20
Mar 20 01:58:38 cheezum kernel:  [<c04707a7>] ? ___ratelimit+0x97/0x110
Mar 20 01:58:38 cheezum kernel:  [<c0186b44>] ? __delayacct_freepages_end+0x24/0x30
Mar 20 01:58:38 cheezum kernel:  [<c0195a8e>] ? T.661+0x5e/0x150
Mar 20 01:58:38 cheezum kernel:  [<c0195396>] ? find_lock_task_mm+0x26/0x70
Mar 20 01:58:38 cheezum kernel:  [<c01957a3>] ? oom_badness+0x113/0x120
Mar 20 01:58:38 cheezum kernel:  [<c0195dbc>] ? out_of_memory+0x23c/0x300
Mar 20 01:58:38 cheezum kernel:  [<c0199777>] ? __alloc_pages_nodemask+0x667/0x680
Mar 20 01:58:38 cheezum kernel:  [<c01370b5>] ? dup_task_struct+0x55/0x120
Mar 20 01:58:38 cheezum kernel:  [<c0137bc8>] ? copy_process+0x88/0xbf0
Mar 20 01:58:38 cheezum kernel:  [<c0120139>] ? do_page_fault+0x169/0x3a0
Mar 20 01:58:38 cheezum kernel:  [<c01387b5>] ? do_fork+0x85/0x260
Mar 20 01:58:38 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:58:38 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 01:58:38 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 01:58:38 cheezum kernel:  [<c010fdb0>] ? sys_clone+0x30/0x40
Mar 20 01:58:38 cheezum kernel:  [<c06e1fcd>] ? ptregs_clone+0x15/0x48
Mar 20 01:58:38 cheezum kernel:  [<c06e16b9>] ? syscall_call+0x7/0xb
Mar 20 01:58:38 cheezum kernel: Mem-Info:
Mar 20 01:58:38 cheezum kernel: DMA per-cpu:
Mar 20 01:58:38 cheezum kernel: CPU    0: hi:    0, btch:   1 usd:   0
Mar 20 01:58:38 cheezum kernel: CPU    1: hi:    0, btch:   1 usd:   0
Mar 20 01:58:38 cheezum kernel: CPU    2: hi:    0, btch:   1 usd:   0
Mar 20 01:58:38 cheezum kernel: CPU    3: hi:    0, btch:   1 usd:   0
Mar 20 01:58:38 cheezum kernel: Normal per-cpu:
Mar 20 01:58:38 cheezum kernel: CPU    0: hi:  186, btch:  31 usd:   0
Mar 20 01:58:38 cheezum kernel: CPU    1: hi:  186, btch:  31 usd:   0
Mar 20 01:58:38 cheezum kernel: CPU    2: hi:  186, btch:  31 usd:  98
Mar 20 01:58:38 cheezum kernel: CPU    3: hi:  186, btch:  31 usd:  90
Mar 20 01:58:38 cheezum kernel: HighMem per-cpu:
Mar 20 01:58:38 cheezum kernel: CPU    0: hi:   18, btch:   3 usd:   0
Mar 20 01:58:38 cheezum kernel: CPU    1: hi:   18, btch:   3 usd:   0
Mar 20 01:58:38 cheezum kernel: CPU    2: hi:   18, btch:   3 usd:   2
Mar 20 01:58:38 cheezum kernel: CPU    3: hi:   18, btch:   3 usd:  15
Mar 20 01:58:38 cheezum kernel: active_anon:1436 inactive_anon:3474 isolated_anon:0
Mar 20 01:58:38 cheezum kernel:  active_file:1 inactive_file:1100 isolated_file:0
Mar 20 01:58:38 cheezum kernel:  unevictable:0 dirty:7 writeback:30 unstable:0
Mar 20 01:58:38 cheezum kernel:  free:6873 slab_reclaimable:10717 slab_unreclaimable:163621
Mar 20 01:58:38 cheezum kernel:  mapped:202 shmem:0 pagetables:584 bounce:0
Mar 20 01:58:38 cheezum kernel: DMA free:2884kB min:72kB low:88kB high:108kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15808kB mlocked:0kB dirty:0kB writeback:4kB mapped:0kB shmem:0kB slab_reclaimable:68kB slab_unreclaimable:3912kB kernel_stack:104kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Mar 20 01:58:38 cheezum kernel: lowmem_reserve[]: 0 700 754 754
Mar 20 01:58:38 cheezum kernel: Normal free:9828kB min:3348kB low:4184kB high:5020kB active_anon:0kB inactive_anon:40kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:717288kB mlocked:0kB dirty:28kB writeback:92kB mapped:0kB shmem:0kB slab_reclaimable:42800kB slab_unreclaimable:650572kB kernel_stack:2776kB pagetables:2336kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:935113 all_unreclaimable? yes
Mar 20 01:58:38 cheezum kernel: lowmem_reserve[]: 0 0 428 428
Mar 20 01:58:38 cheezum kernel: HighMem free:14780kB min:128kB low:192kB high:256kB active_anon:5744kB inactive_anon:13856kB active_file:4kB inactive_file:4400kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:54868kB mlocked:0kB dirty:0kB writeback:24kB mapped:808kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Mar 20 01:58:38 cheezum kernel: lowmem_reserve[]: 0 0 0 0
Mar 20 01:58:38 cheezum kernel: DMA: 213*4kB 218*8kB 18*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2884kB
Mar 20 01:58:38 cheezum kernel: Normal: 2468*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 9872kB
Mar 20 01:58:38 cheezum kernel: HighMem: 894*4kB 976*8kB 179*16kB 15*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 14792kB
Mar 20 01:58:38 cheezum kernel: 3178 total pagecache pages
Mar 20 01:58:38 cheezum kernel: 2037 pages in swap cache
Mar 20 01:58:38 cheezum kernel: Swap cache stats: add 6260556, delete 6258519, find 2696024/2988123
Mar 20 01:58:38 cheezum kernel: Free swap  = 98660kB
Mar 20 01:58:38 cheezum kernel: Total swap = 262140kB
Mar 20 01:58:38 cheezum kernel: 198640 pages RAM
Mar 20 01:58:38 cheezum kernel: 13826 pages HighMem
Mar 20 01:58:38 cheezum kernel: 6538 pages reserved
Mar 20 01:58:38 cheezum kernel: 532 pages shared
Mar 20 01:58:38 cheezum kernel: 183880 pages non-shared
Mar 20 01:58:38 cheezum kernel: [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Mar 20 01:58:38 cheezum kernel: [ 1121]     0  1121      569        0   1     -17         -1000 udevd
Mar 20 01:58:38 cheezum kernel: [ 2613]     0  2613      656        0   1       0             0 login
Mar 20 01:58:38 cheezum kernel: [ 2650]   106  2650      497       14   2       0             0 syslogd
Mar 20 01:58:38 cheezum kernel: [ 2671]     0  2671      481        4   3       0             0 dd
Mar 20 01:58:38 cheezum kernel: [ 2673]   105  2673      631        7   2       0             0 klogd
Mar 20 01:58:38 cheezum kernel: [ 2691]   101  2691      872        5   2       0             0 dnsmasq
Mar 20 01:58:38 cheezum kernel: [ 2712]     0  2712     1342        6   1     -17         -1000 sshd
Mar 20 01:58:38 cheezum kernel: [ 2771]  1000  2771     3085      431   3     -17         -1000 screen
Mar 20 01:58:38 cheezum kernel: [ 2788]  1000  2788     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2804]  1000  2804     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2820]  1000  2820     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2836]  1000  2836     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2858]  1000  2858     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2887]  1000  2887     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2905]  1000  2905     1451        0   3     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2965]  1000  2965     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2987]  1000  2987     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3026]  1000  3026     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3054]  1000  3054     1451        0   2     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3109]     0  3109      539        8   2       0             0 cron
Mar 20 01:58:38 cheezum kernel: [ 3146]     1  3146      441        9   2       0             0 uptimed
Mar 20 01:58:38 cheezum kernel: [ 3189]  1000  3189    13778      782   3     -17         -1000 irssi
Mar 20 01:58:38 cheezum kernel: [ 3203]  1001  3203     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3238]  1000  3238     1450        0   1       0             0 bash
Mar 20 01:58:38 cheezum kernel: [ 3453]  1000  3453     1451        0   3     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3652]     0  3652      786        7   2     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 3656]     0  3656      786        8   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 3658]     0  3658      786        6   0     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 3696]     0  3696     1816      101   3     -17         -1000 tcpdump
Mar 20 01:58:38 cheezum kernel: [12409]  1001 12409     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [12430]  1001 12430     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15137]  1000 15137     1451        0   2     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15154]  1000 15154      823        0   0     -17         -1000 less
Mar 20 01:58:38 cheezum kernel: [15589]  1000 15589     1451        0   3     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15606]  1000 15606      636      114   2     -17         -1000 htop
Mar 20 01:58:38 cheezum kernel: [15607]  1000 15607     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15623]  1000 15623     1219        0   0     -17         -1000 python
Mar 20 01:58:38 cheezum kernel: [15710]  1000 15710     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15726]     0 15726     2046      315   2     -17         -1000 iotop
Mar 20 01:58:38 cheezum kernel: [15727]  1000 15727     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15743]  1000 15743      473       12   3     -17         -1000 vmstat
Mar 20 01:58:38 cheezum kernel: [15760]  1000 15760     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15776]  1000 15776      516       86   3     -17         -1000 iostat
Mar 20 01:58:38 cheezum kernel: [15825]  1000 15825     1451        0   2     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15841]     0 15841     2619       37   2     -17         -1000 tcpdump
Mar 20 01:58:38 cheezum kernel: [15867]  1000 15867     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15883]  1000 15883     4229        3   0     -17         -1000 nano
Mar 20 01:58:38 cheezum kernel: [16026]  1000 16026     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [16050]  1000 16050     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [16214]     0 16214      786        9   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [16660]     0 16660      786        7   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 7028]  1000  7028     1451        1   3     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [14552]  1000 14552     1451        0   2     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [16812]   111 16812     2713        0   1     -17         -1000 nsd
Mar 20 01:58:38 cheezum kernel: [19295]  1000 19295      823        0   3     -17         -1000 less
Mar 20 01:58:38 cheezum kernel: [32143]  1000 32143     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [32190]  1000 32190      823        0   1     -17         -1000 less
Mar 20 01:58:38 cheezum kernel: [ 4050]  1000  4050      786        8   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 1839]  1000  1839     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 1873]  1000  1873      823        0   1     -17         -1000 less
Mar 20 01:58:38 cheezum kernel: [31259]   112 31259    34250        0   1     -17         -1000 apt-cacher-ng
Mar 20 01:58:38 cheezum kernel: [13773]   111 13773     1048        0   3     -17         -1000 nsd
Mar 20 01:58:38 cheezum kernel: [13774]   111 13774     1192        0   2     -17         -1000 nsd
Mar 20 01:58:38 cheezum kernel: [31147]  1000 31147      474       59   3       0             0 vmstat
Mar 20 01:58:38 cheezum kernel: [  929]  1000   929     1452        0   2     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [  949]  1000   949      824        0   2     -17         -1000 less
Mar 20 01:58:38 cheezum kernel: [ 1021]    33  1021     1826      107   2       0             0 lighttpd
Mar 20 01:58:38 cheezum kernel: [ 1024]    33  1024    11996       60   2       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1025]    33  1025    11996       57   2       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1026]    33  1026    11996       54   3       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1027]    33  1027     1619       69   0       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1031]    33  1031     1619       83   3       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1036]    33  1036     1619       68   2       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1038]    33  1038     1865       79   2       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1068]    33  1068     1586       65   2       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1089]    33  1089     1586       74   2       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1109]    33  1109     1586       61   3       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1135]    33  1135     1586       68   0       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 3980]  1000  3980     1452        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3998]  1000  3998      972        0   0     -17         -1000 nano
Mar 20 01:58:38 cheezum kernel: [ 4637]  1000  4637     1452        0   3     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 4654]  1000  4654      972        0   0     -17         -1000 nano
Mar 20 01:58:38 cheezum kernel: Out of memory: Kill process 1021 (lighttpd) score 2 or sacrifice child
Mar 20 01:58:38 cheezum kernel: Killed process 1024 (python) total-vm:47984kB, anon-rss:224kB, file-rss:16kB
Mar 20 01:58:38 cheezum kernel: irssi invoked oom-killer: gfp_mask=0xd0, order=1, oom_adj=-17, oom_score_adj=-1000
Mar 20 01:58:38 cheezum kernel: irssi cpuset=/ mems_allowed=0
Mar 20 01:58:38 cheezum kernel: Pid: 3189, comm: irssi Not tainted 3.2.1-linode40 #1
Mar 20 01:58:38 cheezum kernel: Call Trace:
Mar 20 01:58:38 cheezum kernel:  [<c01958fd>] ? T.662+0x7d/0x1b0
Mar 20 01:58:38 cheezum kernel:  [<c01067fb>] ? xen_restore_fl_direct_reloc+0x4/0x4
Mar 20 01:58:38 cheezum kernel:  [<c06e1431>] ? _raw_spin_unlock_irqrestore+0x11/0x20
Mar 20 01:58:38 cheezum kernel:  [<c04707a7>] ? ___ratelimit+0x97/0x110
Mar 20 01:58:38 cheezum kernel:  [<c0186b44>] ? __delayacct_freepages_end+0x24/0x30
Mar 20 01:58:38 cheezum kernel:  [<c0195a8e>] ? T.661+0x5e/0x150
Mar 20 01:58:38 cheezum kernel:  [<c0195396>] ? find_lock_task_mm+0x26/0x70
Mar 20 01:58:38 cheezum kernel:  [<c01957a3>] ? oom_badness+0x113/0x120
Mar 20 01:58:38 cheezum kernel:  [<c0195dbc>] ? out_of_memory+0x23c/0x300
Mar 20 01:58:38 cheezum kernel:  [<c0199777>] ? __alloc_pages_nodemask+0x667/0x680
Mar 20 01:58:38 cheezum kernel:  [<c01370b5>] ? dup_task_struct+0x55/0x120
Mar 20 01:58:38 cheezum kernel:  [<c0137bc8>] ? copy_process+0x88/0xbf0
Mar 20 01:58:38 cheezum kernel:  [<c0120139>] ? do_page_fault+0x169/0x3a0
Mar 20 01:58:38 cheezum kernel:  [<c01387b5>] ? do_fork+0x85/0x260
Mar 20 01:58:38 cheezum kernel:  [<c010fdb0>] ? sys_clone+0x30/0x40
Mar 20 01:58:38 cheezum kernel:  [<c06e1fcd>] ? ptregs_clone+0x15/0x48
Mar 20 01:58:38 cheezum kernel:  [<c06e16b9>] ? syscall_call+0x7/0xb
Mar 20 01:58:38 cheezum kernel:  [<c06e0000>] ? schedule_timeout_uninterruptible+0x10/0x20
Mar 20 01:58:38 cheezum kernel: Mem-Info:
Mar 20 01:58:38 cheezum kernel: DMA per-cpu:
Mar 20 01:58:38 cheezum kernel: CPU    0: hi:    0, btch:   1 usd:   0
Mar 20 01:58:38 cheezum kernel: CPU    1: hi:    0, btch:   1 usd:   0
Mar 20 01:58:38 cheezum kernel: CPU    2: hi:    0, btch:   1 usd:   0
Mar 20 01:58:38 cheezum kernel: CPU    3: hi:    0, btch:   1 usd:   0
Mar 20 01:58:38 cheezum kernel: Normal per-cpu:
Mar 20 01:58:38 cheezum kernel: CPU    0: hi:  186, btch:  31 usd:   0
Mar 20 01:58:38 cheezum kernel: CPU    1: hi:  186, btch:  31 usd:   0
Mar 20 01:58:38 cheezum kernel: CPU    2: hi:  186, btch:  31 usd: 110
Mar 20 01:58:38 cheezum kernel: CPU    3: hi:  186, btch:  31 usd: 106
Mar 20 01:58:38 cheezum kernel: HighMem per-cpu:
Mar 20 01:58:38 cheezum kernel: CPU    0: hi:   18, btch:   3 usd:   0
Mar 20 01:58:38 cheezum kernel: CPU    1: hi:   18, btch:   3 usd:   0
Mar 20 01:58:38 cheezum kernel: CPU    2: hi:   18, btch:   3 usd:  16
Mar 20 01:58:38 cheezum kernel: CPU    3: hi:   18, btch:   3 usd:   4
Mar 20 01:58:38 cheezum kernel: active_anon:1376 inactive_anon:3409 isolated_anon:0
Mar 20 01:58:38 cheezum kernel:  active_file:1 inactive_file:1155 isolated_file:0
Mar 20 01:58:38 cheezum kernel:  unevictable:0 dirty:7 writeback:30 unstable:0
Mar 20 01:58:38 cheezum kernel:  free:6939 slab_reclaimable:10717 slab_unreclaimable:163621
Mar 20 01:58:38 cheezum kernel:  mapped:272 shmem:0 pagetables:584 bounce:0
Mar 20 01:58:38 cheezum kernel: DMA free:2884kB min:72kB low:88kB high:108kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15808kB mlocked:0kB dirty:0kB writeback:4kB mapped:0kB shmem:0kB slab_reclaimable:68kB slab_unreclaimable:3912kB kernel_stack:104kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Mar 20 01:58:38 cheezum kernel: lowmem_reserve[]: 0 700 754 754
Mar 20 01:58:38 cheezum kernel: Normal free:9828kB min:3348kB low:4184kB high:5020kB active_anon:0kB inactive_anon:40kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:717288kB mlocked:0kB dirty:28kB writeback:92kB mapped:0kB shmem:0kB slab_reclaimable:42800kB slab_unreclaimable:650572kB kernel_stack:2776kB pagetables:2336kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:971290 all_unreclaimable? yes
Mar 20 01:58:38 cheezum kernel: lowmem_reserve[]: 0 0 428 428
Mar 20 01:58:38 cheezum kernel: HighMem free:15044kB min:128kB low:192kB high:256kB active_anon:5504kB inactive_anon:13596kB active_file:4kB inactive_file:4620kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:54868kB mlocked:0kB dirty:0kB writeback:24kB mapped:1088kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Mar 20 01:58:38 cheezum kernel: lowmem_reserve[]: 0 0 0 0
Mar 20 01:58:38 cheezum kernel: DMA: 213*4kB 218*8kB 18*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2884kB
Mar 20 01:58:38 cheezum kernel: Normal: 2468*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 9872kB
Mar 20 01:58:38 cheezum kernel: HighMem: 900*4kB 972*8kB 196*16kB 15*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 15056kB
Mar 20 01:58:38 cheezum kernel: 3227 total pagecache pages
Mar 20 01:58:38 cheezum kernel: 2032 pages in swap cache
Mar 20 01:58:38 cheezum kernel: Swap cache stats: add 6260592, delete 6258560, find 2696047/2988164
Mar 20 01:58:38 cheezum kernel: Free swap  = 100672kB
Mar 20 01:58:38 cheezum kernel: Total swap = 262140kB
Mar 20 01:58:38 cheezum kernel: 198640 pages RAM
Mar 20 01:58:38 cheezum kernel: 13826 pages HighMem
Mar 20 01:58:38 cheezum kernel: 6538 pages reserved
Mar 20 01:58:38 cheezum kernel: 678 pages shared
Mar 20 01:58:38 cheezum kernel: 183801 pages non-shared
Mar 20 01:58:38 cheezum kernel: [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Mar 20 01:58:38 cheezum kernel: [ 1121]     0  1121      569        0   1     -17         -1000 udevd
Mar 20 01:58:38 cheezum kernel: [ 2613]     0  2613      656        0   1       0             0 login
Mar 20 01:58:38 cheezum kernel: [ 2650]   106  2650      497       14   2       0             0 syslogd
Mar 20 01:58:38 cheezum kernel: [ 2671]     0  2671      481        4   3       0             0 dd
Mar 20 01:58:38 cheezum kernel: [ 2673]   105  2673      631       16   3       0             0 klogd
Mar 20 01:58:38 cheezum kernel: [ 2691]   101  2691      872        5   2       0             0 dnsmasq
Mar 20 01:58:38 cheezum kernel: [ 2712]     0  2712     1342        6   1     -17         -1000 sshd
Mar 20 01:58:38 cheezum kernel: [ 2771]  1000  2771     3085      431   3     -17         -1000 screen
Mar 20 01:58:38 cheezum kernel: [ 2788]  1000  2788     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2804]  1000  2804     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2820]  1000  2820     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2836]  1000  2836     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2858]  1000  2858     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2887]  1000  2887     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2905]  1000  2905     1451        0   3     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2965]  1000  2965     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2987]  1000  2987     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3026]  1000  3026     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3054]  1000  3054     1451        0   2     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3109]     0  3109      539        8   2       0             0 cron
Mar 20 01:58:38 cheezum kernel: [ 3146]     1  3146      441        9   2       0             0 uptimed
Mar 20 01:58:38 cheezum kernel: [ 3189]  1000  3189    13778      782   3     -17         -1000 irssi
Mar 20 01:58:38 cheezum kernel: [ 3203]  1001  3203     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3238]  1000  3238     1450        0   1       0             0 bash
Mar 20 01:58:38 cheezum kernel: [ 3453]  1000  3453     1451        0   3     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3652]     0  3652      786        7   2     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 3656]     0  3656      786        8   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 3658]     0  3658      786        6   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 3696]     0  3696     1816      101   3     -17         -1000 tcpdump
Mar 20 01:58:38 cheezum kernel: [12409]  1001 12409     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [12430]  1001 12430     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15137]  1000 15137     1451        0   2     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15154]  1000 15154      823        0   0     -17         -1000 less
Mar 20 01:58:38 cheezum kernel: [15589]  1000 15589     1451        0   3     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15606]  1000 15606      636      143   2     -17         -1000 htop
Mar 20 01:58:38 cheezum kernel: [15607]  1000 15607     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15623]  1000 15623     1219        0   0     -17         -1000 python
Mar 20 01:58:38 cheezum kernel: [15710]  1000 15710     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15726]     0 15726     2046      352   0     -17         -1000 iotop
Mar 20 01:58:38 cheezum kernel: [15727]  1000 15727     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15743]  1000 15743      473       12   3     -17         -1000 vmstat
Mar 20 01:58:38 cheezum kernel: [15760]  1000 15760     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15776]  1000 15776      516       86   3     -17         -1000 iostat
Mar 20 01:58:38 cheezum kernel: [15825]  1000 15825     1451        0   2     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15841]     0 15841     2619       37   2     -17         -1000 tcpdump
Mar 20 01:58:38 cheezum kernel: [15867]  1000 15867     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15883]  1000 15883     4229        3   0     -17         -1000 nano
Mar 20 01:58:38 cheezum kernel: [16026]  1000 16026     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [16050]  1000 16050     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [16214]     0 16214      786        9   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [16660]     0 16660      786        7   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 7028]  1000  7028     1451        1   3     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [14552]  1000 14552     1451        0   2     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [16812]   111 16812     2713        0   1     -17         -1000 nsd
Mar 20 01:58:38 cheezum kernel: [19295]  1000 19295      823        0   3     -17         -1000 less
Mar 20 01:58:38 cheezum kernel: [32143]  1000 32143     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [32190]  1000 32190      823        0   1     -17         -1000 less
Mar 20 01:58:38 cheezum kernel: [ 4050]  1000  4050      786        8   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 1839]  1000  1839     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 1873]  1000  1873      823        0   1     -17         -1000 less
Mar 20 01:58:38 cheezum kernel: [31259]   112 31259    34250        0   1     -17         -1000 apt-cacher-ng
Mar 20 01:58:38 cheezum kernel: [13773]   111 13773     1048        0   3     -17         -1000 nsd
Mar 20 01:58:38 cheezum kernel: [13774]   111 13774     1192        0   2     -17         -1000 nsd
Mar 20 01:58:38 cheezum kernel: [31147]  1000 31147      474       59   3       0             0 vmstat
Mar 20 01:58:38 cheezum kernel: [  929]  1000   929     1452        0   2     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [  949]  1000   949      824        0   2     -17         -1000 less
Mar 20 01:58:38 cheezum kernel: [ 1021]    33  1021     1826      107   2       0             0 lighttpd
Mar 20 01:58:38 cheezum kernel: [ 1025]    33  1025    11996       57   2       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1026]    33  1026    11996       54   3       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1027]    33  1027     1619       69   3       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1031]    33  1031     1619       83   3       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1036]    33  1036    3     4t/l>_x1 MPCu  itB_lBkk Bda(:m0B 0843*4kB 972*8kB 196*16kB 15*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 14828kB
Mar 20 01:58:38 cheezum kernel: 3283 total pagecache pages
Mar 20 01:58:38 cheezum kernel: 2037 pages in swap cache
Mar 20 01:58:38 cheezum kernel: Swap cache stats: add 6260616, delete 6258579, find 2696070/2988198
Mar 20 01:58:38 cheezum kernel: Free swap  = 100748kB
Mar 20 01:58:38 cheezum kernel: Total swap = 262140kB
Mar 20 01:58:38 cheezum kernel: 198640 pages RAM
Mar 20 01:58:38 cheezum kernel: 13826 pages HighMem
Mar 20 01:58:38 cheezum kernel: 6538 pages reserved
Mar 20 01:58:38 cheezum kernel: 790 pages shared
Mar 20 01:58:38 cheezum kernel: 183859 pages non-shared
Mar 20 01:58:38 cheezum kernel: [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Mar 20 01:58:38 cheezum kernel: [ 1121]     0  1121      569        0   1     -17         -1000 udevd
Mar 20 01:58:38 cheezum kernel: [ 2613]     0  2613      656        0   1       0             0 login
Mar 20 01:58:38 cheezum kernel: [ 2650]   106  2650      497       50   2       0             0 syslogd
Mar 20 01:58:38 cheezum kernel: [ 2671]     0  2671      481        4   3       0             0 dd
Mar 20 01:58:38 cheezum kernel: [ 2673]   105  2673      631       16   2       0             0 klogd
Mar 20 01:58:38 cheezum kernel: [ 2691]   101  2691      872        5   2       0             0 dnsmasq
Mar 20 01:58:38 cheezum kernel: [ 2712]     0  2712     1342        6   1     -17         -1000 sshd
Mar 20 01:58:38 cheezum kernel: [ 2771]  1000  2771     3085      431   3     -17         -1000 screen
Mar 20 01:58:38 cheezum kernel: [ 2788]  1000  2788     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2804]  1000  2804     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2820]  1000  2820     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2836]  1000  2836     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2858]  1000  2858     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2887]  1000  2887     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2905]  1000  2905     1451        0   3     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2965]  1000  2965     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2987]  1000  2987     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3026]  1000  3026     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3054]  1000  3054     1451        0   2     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3109]     0  3109      539        8   2       0             0 cron
Mar 20 01:58:38 cheezum kernel: [ 3146]     1  3146      441        9   2       0             0 uptimed
Mar 20 01:58:38 cheezum kernel: [ 3189]  1000  3189    13778      782   2     -17         -1000 irssi
Mar 20 01:58:38 cheezum kernel: [ 3203]  1001  3203     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3238]  1000  3238     1450        0   1       0             0 bash
Mar 20 01:58:38 cheezum kernel: [ 3453]  1000  3453     1451        0   3     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3652]     0  3652      786        7   2     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 3656]     0  3656      786        8   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 3658]     0  3658      786        6   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 3696]     0  3696     1816      101   3     -17         -1000 tcpdump
Mar 20 01:58:38 cheezum kernel: [12409]  1001 12409     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [12430]  1001 12430     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15137]  1000 15137     1451        0   2     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15154]  1000 15154      823        0   0     -17         -1000 less
Mar 20 01:58:38 cheezum kernel: [15589]  1000 15589     1451        0   3     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15606]  1000 15606      636      143   2     -17         -1000 htop
Mar 20 01:58:38 cheezum kernel: [15607]  1000 15607     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15623]  1000 15623     1219        0   0     -17         -1000 python
Mar 20 01:58:38 cheezum kernel: [15710]  1000 15710     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15726]     0 15726     2046      352   0     -17         -1000 iotop
Mar 20 01:58:38 cheezum kernel: [15727]  1000 15727     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15743]  1000 15743      473       12   3     -17         -1000 vmstat
Mar 20 01:58:38 cheezum kernel: [15760]  1000 15760     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15776]  1000 15776      516       86   3     -17         -1000 iostat
Mar 20 01:58:38 cheezum kernel: [15825]  1000 15825     1451        0   2     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15841]     0 15841     2619       37   2     -17         -1000 tcpdump
Mar 20 01:58:38 cheezum kernel: [15867]  1000 15867     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15883]  1000 15883     4229        3   0     -17         -1000 nano
Mar 20 01:58:38 cheezum kernel: [16026]  1000 16026     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [16050]  1000 16050     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [16214]     0 16214      786        9   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [16660]     0 16660      786        7   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 7028]  1000  7028     1451        1   3     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [14552]  1000 14552     1451        0   2     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [16812]   111 16812     2713        0   1     -17         -1000 nsd
Mar 20 01:58:38 cheezum kernel: [19295]  1000 19295      823        0   3     -17         -1000 less
Mar 20 01:58:38 cheezum kernel: [32143]  1000 32143     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [32190]  1000 32190      823        0   1     -17         -1000 less
Mar 20 01:58:38 cheezum kernel: [ 4050]  1000  4050      786        8   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 1839]  1000  1839     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 1873]  1000  1873      823        0   1     -17         -1000 less
Mar 20 01:58:38 cheezum kernel: [31259]   112 31259    34250        0   1     -17         -1000 apt-cacher-ng
Mar 20 01:58:38 cheezum kernel: [13773]   111 13773     1048        0   3     -17         -1000 nsd
Mar 20 01:58:38 cheezum kernel: [13774]   111 13774     1192        0   2     -17         -1000 nsd
Mar 20 01:58:38 cheezum kernel: [31147]  1000 31147      474       59   3       0             0 vmstat
Mar 20 01:58:38 cheezum kernel: [  929]  1000   929     1452        0   2     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [  949]  1000   949      824        0   2     -17         -1000 less
Mar 20 01:58:38 cheezum kernel: [ 1021]    33  1021     1826      107   2       0             0 lighttpd
Mar 20 01:58:38 cheezum kernel: [ 1058]    33  1025    11996       75   0       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1026]    33  1026    11996       54   3       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1027]    33  1027     1619      119   2       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1031]    33  1031     1619       83   3       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1036]    33  1036     1619       68   2       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1038]    33  1038     1865      109   3       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1068]    33  1068     1586       65   0       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1089]    33  1089     1586       74   2       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1109]    33  1109     1586       61   3       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1135]    33  1135     1586       78   3       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 3980]  1000  3980     1452        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3998]  1000  3998      972        0   0     -17         -1000 nano
Mar 20 01:58:38 cheezum kernel: [ 4637]  1000  4637     1452        0   3     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 4654]  1000  4654      972        0   0     -17         -1000 nano
Mar 20 01:58:38 cheezum kernel: Out of memory: Kill process 1021 (lighttpd) score 2 or sacrifice child
Mar 20 01:58:38 cheezum kernel: Killed process 1058 (python) total-vm:47984kB, anon-rss:244kB, file-rss:56kB
Mar 20 01:58:38 cheezum kernel: irssi invoked oom-killer: gfp_mask=0xd0, order=1, oom_adj=-17, oom_score_adj=-1000
Mar 20 01:58:38 cheezum kernel: irssi cpuset=/ mems_allowed=0
Mar 20 01:58:38 cheezum kernel: Pid: 3189, comm: irssi Not tainted 3.2.1-linode40 #1
Mar 20 01:58:38 cheezum kernel: Call Trace:
Mar 20 01:58:38 cheezum kernel:  [<c01958fd>] ? T.662+0x7d/0x1b0
Mar 20 01:58:38 cheezum kernel:  [<c01067fb>] ? xen_restore_fl_direct_reloc+0x4/0x4
Mar 20 01:58:38 cheezum kernel:  [<c06e1431>] ? _raw_spin_unlock_irqrestore+0x11/0x20
Mar 20 01:58:38 cheezum kernel:  [<c04707a7>] ? ___ratelimit+0x97/0x110
Mar 20 01:58:38 cheezum kernel:  [<c0186b44>] ? __delayacct_freepages_end+0x24/0x30
Mar 20 01:58:38 cheezum kernel:  [<c0195a8e>] ? T.661+0x5e/0x150
Mar 20 01:58:38 cheezum kernel:  [<c0195396>] ? find_lock_task_mm+0x26/0x70
Mar 20 01:58:38 cheezum kernel:  [<c01957a3>] ? oom_badness+0x113/0x120
Mar 20 01:58:38 cheezum kernel:  [<c0195dbc>] ? out_of_memory+0x23c/0x300
Mar 20 01:58:38 cheezum kernel:  [<c0199777>] ? __alloc_pages_nodemask+0x667/0x680
Mar 20 01:58:38 cheezum kernel:  [<c01370b5>] ? dup_task_struct+0x55/0x120
Mar 20 01:58:38 cheezum kernel:  [<c0137bc8>] ? copy_process+0x88/0xbf0
Mar 20 01:58:38 cheezum kernel:  [<c0120139>] ? do_page_fault+0x169/0x3a0
Mar 20 01:58:38 cheezum kernel:  [<c01387b5>] ? do_fork+0x85/0x260
Mar 20 01:58:38 cheezum kernel:  [<c010fdb0>] ? sys_clone+0x30/0x40
Mar 20 01:58:38 cheezum kernel:  [<c06e1fcd>] ? ptregs_clone+0x15/0x48
Mar 20 01:58:38 cheezum kernel:  [<c06e16b9>] ? syscall_call+0x7/0xb
Mar 20 01:58:38 cheezum kernel:  [<c06e0000>] ? schedule_timeout_uninterruptible+0x10/0x20
Mar 20 01:58:38 cheezum kernel: Mem-Info:
Mar 20 01:58:38 cheezum kernel: DMA per-cpu:
Mar 20 01:58:38 cheezum kernel: CPU    0: hi:    0, btch:   1 usd:   0
Mar 20 01:58:38 cheezum kernel: CPU    1: hi:    0, btch:   1 usd:   0
Mar 20 01:58:38 cheezum kernel: CPU    2: hi:    0, btch:   1 usd:   0
Mar 20 01:58:38 cheezum kernel: CPU    3: hi:    0, btch:   1 usd:   0
Mar 20 01:58:38 cheezum kernel: Normal per-cpu:
Mar 20 01:58:38 cheezum kernel: CPU    0: hi:  186, btch:  31 usd:   0
Mar 20 01:58:38 cheezum kernel: CPU    1: hi:  186, btch:  31 usd:   0
Mar 20 01:58:38 cheezum kernel: CPU    2: hi:  186, btch:  31 usd: 123
Mar 20 01:58:38 cheezum kernel: CPU    3: hi:  186, btch:  31 usd: 108
Mar 20 01:58:38 cheezum kernel: HighMem per-cpu:
Mar 20 01:58:38 cheezum kernel: CPU    0: hi:   18, btch:   3 usd:   0
Mar 20 01:58:38 cheezum kernel: CPU    1: hi:   18, btch:   3 usd:   0
Mar 20 01:58:38 cheezum kernel: CPU    2: hi:   18, btch:   3 usd:  15
Mar 20 01:58:38 cheezum kernel: CPU    3: hi:   18, btch:   3 usd:   0
Mar 20 01:58:38 cheezum kernel: active_anon:1356 inactive_anon:3414 isolated_anon:0
Mar 20 01:58:38 cheezum kernel:  active_file:33 inactive_file:1238 isolated_file:0
Mar 20 01:58:38 cheezum kernel:  unevictable:0 dirty:7 writeback:30 unstable:0
Mar 20 01:58:38 cheezum kernel:  free:6891 slab_reclaimable:10717 slab_unreclaimable:163621
Mar 20 01:58:38 cheezum kernel:  mapped:300 shmem:0 pagetables:584 bounce:0
Mar 20 01:58:38 cheezum kernel: DMA free:2884kB min:72kB low:88kB high:108kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15808kB mlocked:0kB dirty:0kB writeback:4kB mapped:0kB shmem:0kB slab_reclaimable:68kB slab_unreclaimable:3912kB kernel_stack:104kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Mar 20 01:58:38 cheezum kernel: lowmem_reserve[]: 0 700 754 754
Mar 20 01:58:38 cheezum kernel: Normal free:9828kB min:3348kB low:4184kB high:5020kB active_anon:0kB inactive_anon:40kB active_file:88kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:717288kB mlocked:0kB dirty:28kB writeback:92kB mapped:0kB shmem:0kB slab_reclaimable:42800kB slab_unreclaimable:650572kB kernel_stack:2776kB pagetables:2336kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:992829 all_unreclaimable? yes
Mar 20 01:58:38 cheezum kernel: lowmem_reserve[]: 0 0 428 428
Mar 20 01:58:38 cheezum kernel: HighMem free:14852kB min:128kB low:192kB high:256kB active_anon:5424kB inactive_anon:13616kB active_file:44kB inactive_file:4960kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:54868kB mlocked:0kB dirty:0kB writeback:24kB mapped:1200kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Mar 20 01:58:38 cheezum kernel: lowmem_reserve[]: 0 0 0 0
Mar 20 01:58:38 cheezum kernel: DMA: 213*4kB 218*8kB 18*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2884kB
Mar 20 01:58:38 cheezum kernel: Normal: 2468*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 9872kB
Mar 20 01:58:38 cheezum kernel: HighMem: 821*4kB 978*8kB 200*16kB 15*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 14852kB
Mar 20 01:58:38 cheezum kernel: 3304 total pagecache pages
Mar 20 01:58:38 cheezum kernel: 2016 pages in swap cache
Mar 20 01:58:38 cheezum kernel: Swap cache stats: add 6260659, delete 6258644, find 2696113/2988262
Mar 20 01:58:38 cheezum kernel: Free swap  = 102852kB
Mar 20 01:58:38 cheezum kernel: Total swap = 262140kB
Mar 20 01:58:38 cheezum kernel: 198640 pages RAM
Mar 20 01:58:38 cheezum kernel: 13826 pages HighMem
Mar 20 01:58:38 cheezum kernel: 6538 pages reserved
Mar 20 01:58:38 cheezum kernel: 952 pages shared
Mar 20 01:58:38 cheezum kernel: 183816 pages non-shared
Mar 20 01:58:38 cheezum kernel: [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Mar 20 01:58:38 cheezum kernel: [ 1121]     0  1121      569        0   1     -17         -1000 udevd
Mar 20 01:58:38 cheezum kernel: [ 2613]     0  2613      656        0   1       0             0 login
Mar 20 01:58:38 cheezum kernel: [ 2650]   106  2650      497       50   2       0             0 syslogd
Mar 20 01:58:38 cheezum kernel: [ 2671]     0  2671      481        4   3       0             0 dd
Mar 20 01:58:38 cheezum kernel: [ 2673]   105  2673      631       16   2       0             0 klogd
Mar 20 01:58:38 cheezum kernel: [ 2691]   101  2691      872        5   2       0             0 dnsmasq
Mar 20 01:58:38 cheezum kernel: [ 2712]     0  2712     1342        6   1     -17         -1000 sshd
Mar 20 01:58:38 cheezum kernel: [ 2771]  1000  2771     3085      466   3     -17         -1000 screen
Mar 20 01:58:38 cheezum kernel: [ 2788]  1000  2788     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2804]  1000  2804     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2820]  1000  2820     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2836]  1000  2836     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2858]  1000  2858     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2887]  1000  2887     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2905]  1000  2905     1451        0   3     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2965]  1000  2965     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2987]  1000  2987     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3026]  1000  3026     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3054]  1000  3054     1451        0   2     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3109]     0  3109      539        8   2       0             0 cron
Mar 20 01:58:38 cheezum kernel: [ 3146]     1  3146      441        9   2       0             0 uptimed
Mar 20 01:58:38 cheezum kernel: [ 3189]  1000  3189    13778      782   2     -17         -1000 irssi
Mar 20 01:58:38 cheezum kernel: [ 3203]  1001  3203     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3238]  1000  3238     1450        0   1       0             0 bash
Mar 20 01:58:38 cheezum kernel: [ 3453]  1000  3453     1451        0   3     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3652]     0  3652      786        7   2     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 3656]     0  3656      786        8   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 3658]     0  3658      786        6   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 3696]     0  3696     1816      101   3     -17         -1000 tcpdump
Mar 20 01:58:38 cheezum kernel: [12409]  1001 12409     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [12430]  1001 12430     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15137]  1000 15137     1451        0   2     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15154]  1000 15154      823        0   0     -17         -1000 less
Mar 20 01:58:38 cheezum kernel: [15589]  1000 15589     1451        0   3     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15606]  1000 15606      636      143   2     -17         -1000 htop
Mar 20 01:58:38 cheezum kernel: [15607]  1000 15607     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15623]  1000 15623     1219        0   0     -17         -1000 python
Mar 20 01:58:38 cheezum kernel: [15710]  1000 15710     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15726]     0 15726     2046      352   0     -17         -1000 iotop
Mar 20 01:58:38 cheezum kernel: [15727]  1000 15727     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15743]  1000 15743      473       12   3     -17         -1000 vmstat
Mar 20 01:58:38 cheezum kernel: [15760]  1000 15760     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15776]  1000 15776      516       86   3     -17         -1000 iostat
Mar 20 01:58:38 cheezum kernel: [15825]  1000 15825     1451        0   2     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15841]     0 15841     2619       37   2     -17         -1000 tcpdump
Mar 20 01:58:38 cheezum kernel: [15867]  1000 15867     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15883]  1000 15883     4229        3   0     -17         -1000 nano
Mar 20 01:58:38 cheezum kernel: [16026]  1000 16026     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [16050]  1000 16050     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [16214]     0 16214      786        9   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [16660]     0 16660      786        7   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 7028]  1000  7028     1451        1   3     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [14552]  1000 14552     1451        0   2     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [16812]   111 16812     2713        0   1     -17         -1000 nsd
Mar 20 01:58:38 cheezum kernel: [19295]  1000 19295      823        0   3     -17         -1000 less
Mar 20 01:58:38 cheezum kernel: [32143]  1000 32143     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [32190]  1000 32190      823        0   1     -17         -1000 less
Mar 20 01:58:38 cheezum kernel: [ 4050]  1000  4050      786        8   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 1839]  1000  1839     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 1873]  1000  1873      823        0   1     -17         -1000 less
Mar 20 01:58:38 cheezum kernel: [31259]   112 31259    34250        0   1     -17         -1000 apt-cacher-ng
Mar 20 01:58:38 cheezum kernel: [13773]   111 13773     1048        0   3     -17         -1000 nsd
Mar 20 01:58:38 cheezum kernel: [13774]   111 13774     1192        0   2     -17         -1000 nsd
Mar 20 01:58:38 cheezum kernel: [31147]  1000 31147      474       59   3       0             0 vmstat
Mar 20 01:58:38 cheezum kernel: [  929]  1000   929     1452        0   2     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [  949]  1000   949      824        0   2     -17         -1000 less
Mar 20 01:58:38 cheezum kernel: [ 1021]    33  1021     1826      107   2       0             0 lighttpd
Mar 20 01:58:38 cheezum kernel: [ 1026]    33  1026    11996       98   3       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1027]    33  1027     1619      119   2       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1031]    33  1031     1619       83   3       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1036]    33  1036     1619       68   0       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1038]    33  1038     1865      109   3       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1068]    33  1068     1586       65   0       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1089]    33  1089     1586       74   2       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1109]    33  1109     1586       91   3       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1135]    33  1135     1586       78   3       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 3980]  1000  3980     1452        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3998]  1000  3998      972        0   0     -17         -1000 nano
Mar 20 01:58:38 cheezum kernel: [ 4637]  1000  4637     1452        0   3     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 4654]  1000  4654      972        0   0     -17         -1000 nano
Mar 20 01:58:38 cheezum kernel: Out of memory: Kill process 1021 (lighttpd) score 2 or sacrifice child
Mar 20 01:58:38 cheezum kernel: Killed process 1026 (python) total-vm:47984kB, anon-rss:248kB, file-rss:144kB
Mar 20 01:58:38 cheezum kernel: cron invoked oom-killer: gfp_mask=0xd0, order=1, oom_adj=0, oom_score_adj=0
Mar 20 01:58:38 cheezum kernel: cron cpuset=/ mems_allowed=0
Mar 20 01:58:38 cheezum kernel: Pid: 3109, comm: cron Not tainted 3.2.1-linode40 #1
Mar 20 01:58:38 cheezum kernel: Call Trace:
Mar 20 01:58:38 cheezum kernel:  [<c01958fd>] ? T.662+0x7d/0x1b0
Mar 20 01:58:38 cheezum kernel:  [<c01067fb>] ? xen_restore_fl_direct_reloc+0x4/0x4
Mar 20 01:58:38 cheezum kernel:  [<c06e1431>] ? _raw_spin_unlock_irqrestore+0x11/0x20
Mar 20 01:58:38 cheezum kernel:  [<c04707a7>] ? ___ratelimit+0x97/0x110
Mar 20 01:58:38 cheezum kernel:  [<c0186b44>] ? __delayacct_freepages_end+0x24/0x30
Mar 20 01:58:38 cheezum kernel:  [<c0195a8e>] ? T.661+0x5e/0x150
Mar 20 01:58:38 cheezum kernel:  [<c0195396>] ? find_lock_task_mm+0x26/0x70
Mar 20 01:58:38 cheezum kernel:  [<c01957a3>] ? oom_ 0,tc_kB *6_>     1  1000 15825     1451        0   2-   l>c3krtack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pag8B e  s2 7      50]  1000 15760     1451        0   0     -17         >    17 s4 i0h 84 isolated(file):0kB present:54868kB mlocked:0kB dirty:200kB writeback:0kB mapped:1576kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Mar 20 01:58:38 cheezum kernel: lowmem_reserve[]: 0 0 0 0
Mar 20 01:58:38 cheezum kernel: DMA: 213*4kB 218*8kB 18*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2884kB
Mar 20 01:58:38 cheezum kernel: Normal: 2404*4kB 1*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 9624kB
Mar 20 01:58:38 cheezum kernel: HighMem: 733*4kB 990*8kB 206*16kB 20*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 14852kB
Mar 20 01:58:38 cheezum kernel: 3444 total pagecache pages
Mar 20 01:58:38 cheezum kernel: 2015 pages in swap cache
Mar 20 01:58:38 cheezum kernel: Swap cache stats: add 6260793, delete 6258778, find 2696205/2988403
Mar 20 01:58:38 cheezum kernel: Free swap  = 109048kB
Mar 20 01:58:38 cheezum kernel: Total swap = 262140kB
Mar 20 01:58:38 cheezum kernel: 198640 pages RAM
Mar 20 01:58:38 cheezum kernel: 13826 pages HighMem
Mar 20 01:58:38 cheezum kernel: 6538 pages reserved
Mar 20 01:58:38 cheezum kernel: 1084 pages shared
Mar 20 01:58:38 cheezum kernel: 183644 pages non-shared
Mar 20 01:58:38 cheezum kernel: [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Mar 20 01:58:38 cheezum kernel: [ 1121]     0  1121      569        0   1     -17         -1000 udevd
Mar 20 01:58:38 cheezum kernel: [ 2613]     0  2613      656        0   1       0             0 login
Mar 20 01:58:38 cheezum kernel: [ 2650]   106  2650      497       50   3       0             0 syslogd
Mar 20 01:58:38 cheezum kernel: [ 2671]     0  2671      481        4   2       0             0 dd
Mar 20 01:58:38 cheezum kernel: [ 2673]   105  2673      631       16   2       0             0 klogd
Mar 20 01:58:38 cheezum kernel: [ 2691]   101  2691      872        5   2       0             0 dnsmasq
Mar 20 01:58:38 cheezum kernel: [ 2712]     0  2712     1342        6   1     -17         -1000 sshd
Mar 20 01:58:38 cheezum kernel: [ 2771]  1000  2771     3085      467   3     -17         -1000 screen
Mar 20 01:58:38 cheezum kernel: [ 2788]  1000  2788     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2804]  1000  2804     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2820]  1000  2820     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2836]  1000  2836     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2858]  1000  2858     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2887]  1000  2887     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2905]  1000  2905     1451        0   3     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2965]  1000  2965     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2987]  1000  2987     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3026]  1000  3026     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3054]  1000  3054     1451        0   2     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3109]     0  3109      539        9   3       0             0 cron
Mar 20 01:58:38 cheezum kernel: [ 3146]     1  3146      441       10   2       0             0 uptimed
Mar 20 01:58:38 cheezum kernel: [ 3189]  1000  3189    13778      783   3     -17         -1000 irssi
Mar 20 01:58:38 cheezum kernel: [ 3203]  1001  3203     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3238]  1000  3238     1450        0   1       0             0 bash
Mar 20 01:58:38 cheezum kernel: [ 3453]  1000  3453     1451        0   3     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3652]     0  3652      786        7   2     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 3656]     0  3656      786        8   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 3658]     0  3658      786        6   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 3696]     0  3696     1816      101   3     -17         -1000 tcpdump
Mar 20 01:58:38 cheezum kernel: [12409]  1001 12409     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [12430]  1001 12430     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15137]  1000 15137     1451        0   2     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15154]  1000 15154      823        0   0     -17         -1000 less
Mar 20 01:58:38 cheezum kernel: [15589]  1000 15589     1451        0   3     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15606]  1000 15606      636      144   2     -17         -1000 htop
Mar 20 01:58:38 cheezum kernel: [15607]  1000 15607     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15623]  1000 15623     1219        0   0     -17         -1000 python
Mar 20 01:58:38 cheezum kernel: [15710]  1000 15710     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15726]     0 15726     2046      561   3     -17         -1000 iotop
Mar 20 01:58:38 cheezum kernel: [15727]  1000 15727     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15743]  1000 15743      473       13   3     -17         -1000 vmstat
Mar 20 01:58:38 cheezum kernel: [15760]  1000 15760     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15776]  1000 15776      516       86   3     -17         -1000 iostat
Mar 20 01:58:38 cheezum kernel: [15825]  1000 15825     1451        0   2     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15841]     0 15841     2619       37   2     -17         -1000 tcpdump
Mar 20 01:58:38 cheezum kernel: [15867]  1000 15867     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15883]  1000 15883     4229        3   0     -17         -1000 nano
Mar 20 01:58:38 cheezum kernel: [16026]  1000 16026     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [16050]  1000 16050     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [16214]     0 16214      786        9   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [16660]     0 16660      786        7   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 7028]  1000  7028     1451        1   3     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [14552]  1000 14552     1451        0   2     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [16812]   111 16812     2713        0   1     -17         -1000 nsd
Mar 20 01:58:38 cheezum kernel: [19295]  1000 19295      823        0   3     -17         -1000 less
Mar 20 01:58:38 cheezum kernel: [32143]  1000 32143     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [32190]  1000 32190      823        0   1     -17         -1000 less
Mar 20 01:58:38 cheezum kernel: [ 4050]  1000  4050      786        8   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 1839]  1000  1839     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 1873]  1000  1873      823        0   1     -17         -1000 less
Mar 20 01:58:38 cheezum kernel: [31259]   112 31259    34250        0   1     -17         -1000 apt-cacher-ng
Mar 20 01:58:38 cheezum kernel: [13773]   111 13773     1048        0   3     -17         -1000 nsd
Mar 20 01:58:38 cheezum kernel: [13774]   111 13774     1192        0   2     -17         -1000 nsd
Mar 20 01:58:38 cheezum kernel: [31147]  1000 31147      474       60   3       0             0 vmstat
Mar 20 01:58:38 cheezum kernel: [  929]  1000   929     1452        0   2     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [  949]  1000   949      824        0   2     -17         -1000 less
Mar 20 01:58:38 cheezum kernel: [ 1021]    33  1021     1826      107   2       0             0 lighttpd
Mar 20 01:58:38 cheezum kernel: [ 1031]    33  1031     1619       84   3       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1036]    33  1036     1619       68   0       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1068]    33  1068     1586      120   2       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1089]    33  1089     1586       74   2       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1109]    33  1109     1586       92   3       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 1135]    33  1135     1586       78   3       0             0 python
Mar 20 01:58:38 cheezum kernel: [ 3980]  1000  3980     1452        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3998]  1000  3998      972        0   0     -17         -1000 nano
Mar 20 01:58:38 cheezum kernel: [ 4637]  1000  4637     1452        0   3     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 4654]  1000  4654      972        0   0     -17         -1000 nano
Mar 20 01:58:38 cheezum kernel: Out of memory: Kill process 1021 (lighttpd) score 2 or sacrifice child
Mar 20 01:58:38 cheezum kernel: Killed process 1031 (python) total-vm:6476kB, anon-rss:272kB, file-rss:64kB
Mar 20 01:58:38 cheezum kernel: cron invoked oom-killer: gfp_mask=0xd0, order=1, oom_adj=0, oom_score_adj=0
Mar 20 01:58:38 cheezum kernel: cron cpuset=/ mems_allowed=0
Mar 20 01:58:38 cheezum kernel: Pid: 3109, comm: cron Not tainted 3.2.1-linode40 #1
Mar 20 01:58:38 cheezum kernel: Call Trace:
Mar 20 01:58:38 cheezum kernel:  [<c01958fd>] ? T.662+0x7d/0x1b0
Mar 20 01:58:38 cheezum kernel:  [<c01067fb>] ? xen_restore_fl_direct_reloc+0x4/0x4
Mar 20 01:58:38 cheezum kernel:  [<c06e1431>] ? _raw_spin_unlock_irqrestore+0x11/0x20
Mar 20 01:58:38 cheezum kernel:  [<c04707a7>] ? ___ratelimit+0x97/0x110
Mar 20 01:58:38 cheezum kernel:  [<c0186b44>] ? __delayacct_freepages_end+0x24/0x30
Mar 20 01:58:38 cheezum kernel:  [<c0195a8e>] ? T.661+0x5e/0x150
Mar 20 01:58:38 cheezum kernel:  [<c0195396>] ? find_lock_task_mm+0x26/0x70
Mar 20 01:58:38 cheezum kernel:  [<c01957a3>] ? oom_badness+0x113/0x120
Mar 20 01:58:38 cheezum kernel:  [<c0195dbc>] ? out_of_memory+0x23c/0x300
Mar 20 01:58:38 cheezum kernel:  [<c0199777>] ? __alloc_pages_nodemask+0x667/0x680
Mar 20 01:58:38 cheezum kernel:  [<c01370b5>] ? dup_task_struct+0x55/0x120
Mar 20 01:58:38 cheezum kernel:  [<c0137bc8>] ? copy_process+0x88/0xbf0
Mar 20 01:58:38 cheezum kernel:  [<c0120139>] ? do_page_fault+0x169/0x3a0
Mar 20 01:58:38 cheezum kernel:  [<c01387b5>] ? do_fork+0x85/0x260
Mar 20 01:58:38 cheezum kernel:  [<c0183671>] ? handle_percpu_irq+0x31/0x50
Mar 20 01:58:38 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 01:58:38 cheezum kernel:  [<c0106047>] ? xen_force_evtchn_callback+0x17/0x30
Mar 20 01:58:38 cheezum kernel:  [<c010fdb0>] ? sys_clone+0x30/0x40
Mar 20 01:58:38 cheezum kernel:  [<c06e1fcd>] ? ptregs_clone+0x15/0x48
Mar 20 01:58:38 cheezum kernel:  [<c06e16b9>] ? syscall_call+0x7/0xb
Mar 20 01:58:38 cheezum kernel: Mem-Info:
Mar 20 01:58:38 cheezum kernel: DMA per-cpu:
Mar 20 01:58:38 cheezum kernel: CPU    0: hi:    0, btch:   1 usd:   0
Mar 20 01:58:38 cheezum kernel: CPU    1: hi:    0, btch:   1 usd:   0
Mar 20 01:58:38 cheezum kernel: CPU    2: hi:    0, btch:   1 usd:   0
Mar 20 01:58:38 cheezum kernel: CPU    3: hi:    0, btch:   1 usd:   0
Mar 20 01:58:38 cheezum kernel: Normal per-cpu:
Mar 20 01:58:38 cheezum kernel: CPU    0: hi:  186, btch:  31 usd:   0
Mar 20 01:58:38 cheezum kernel: CPU    1: hi:  186, btch:  31 usd:   0
Mar 20 01:58:38 cheezum kernel: CPU    2: hi:  186, btch:  31 usd: 150
Mar 20 01:58:38 cheezum kernel: CPU    3: hi:  186, btch:  31 usd: 178
Mar 20 01:58:38 cheezum kernel: HighMem per-cpu:
Mar 20 01:58:38 cheezum kernel: CPU    0: hi:   18, btch:   3 usd:   0
Mar 20 01:58:38 cheezum kernel: CPU    1: hi:   18, btch:   3 usd:   0
Mar 20 01:58:38 cheezum kernel: CPU    2: hi:   18, btch:   3 usd:  17
Mar 20 01:58:38 cheezum kernel: CPU    3: hi:   18, btch:   3 usd:  16
Mar 20 01:58:38 cheezum kernel: active_anon:1261 inactive_anon:3274 isolated_anon:0
Mar 20 01:58:38 cheezum kernel:  active_file:76 inactive_file:1323 isolated_file:0
Mar 20 01:58:38 cheezum kernel:  unevictable:0 dirty:57 writeback:23 unstable:0
Mar 20 01:58:38 cheezum kernel:  free:6898 slab_reclaimable:10722 slab_unreclaimable:163626
Mar 20 01:58:38 cheezum kernel:  mapped:394 shmem:0 pagetables:516 bounce:0
Mar 20 01:58:38 cheezum kernel: DMA free:2884kB min:72kB low:88kB high:108kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15808kB mlocked:0kB dirty:0kB writeback:4kB mapped:0kB shmem:0kB slab_reclaimable:68kB slab_unreclaimable:3912kB kernel_stack:104kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Mar 20 01:58:38 cheezum kernel: lowmem_reserve[]: 0 700 754 754
Mar 20 01:58:38 cheezum kernel: Normal free:9580kB min:3348kB low:4184kB high:5020kB active_anon:4kB inactive_anon:24kB active_file:0kB inactive_file:52kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:717288kB mlocked:0kB dirty:28kB writeback:92kB mapped:0kB shmem:0kB slab_reclaimable:42820kB slab_unreclaimable:650592kB kernel_stack:2776kB pagetables:2064kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:155 all_unreclaimable? yes
Mar 20 01:58:38 cheezum kernel: lowmem_reserve[]: 0 0 428 428
Mar 20 01:58:38 cheezum kernel: HighMem free:15128kB min:128kB low:192kB high:256kB active_anon:5040kB inactive_anon:13072kB active_file:312kB inactive_file:5240kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:54868kB mlocked:0kB dirty:200kB writeback:0kB mapped:1576kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Mar 20 01:58:38 cheezum kernel: lowmem_reserve[]: 0 0 0 0
Mar 20 01:58:38 cheezum kernel: DMA: 213*4kB 218*8kB 18*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2884kB
Mar 20 01:58:38 cheezum kernel: Normal: 2404*4kB 1*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 9624kB
Mar 20 01:58:38 cheezum kernel: HighMem: 771*4kB 997*8kB 209*16kB 21*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 15140kB
Mar 20 01:58:38 cheezum kernel: 3430 total pagecache pages
Mar 20 01:58:38 cheezum kernel: 2001 pages in swap cache
Mar 20 01:58:38 cheezum kernel: Swap cache stats: add 6260794, delete 6258793, find 2696207/2988406
Mar 20 01:58:38 cheezum kernel: Free swap  = 110612kB
Mar 20 01:58:38 cheezum kernel: Total swap = 262140kB
Mar 20 01:58:38 cheezum kernel: 198640 pages RAM
Mar 20 01:58:38 cheezum kernel: 13826 pages HighMem
Mar 20 01:58:38 cheezum kernel: 6538 pages reserved
Mar 20 01:58:38 cheezum kernel: 1048 pages shared
Mar 20 01:58:38 cheezum kernel: 183545 pages non-shared
Mar 20 01:58:38 cheezum kernel: [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Mar 20 01:58:38 cheezum kernel: [ 1121]     0  1121      569        0   1     -17         -1000 udevd
Mar 20 01:58:38 cheezum kernel: [ 2613]     0  2613      656        0   1       0             0 login
Mar 20 01:58:38 cheezum kernel: [ 2650]   106  2650      497       50   3       0             0 syslogd
Mar 20 01:58:38 cheezum kernel: [ 2671]     0  2671      481        4   2       0             0 dd
Mar 20 01:58:38 cheezum kernel: [ 2673]   105  2673      631       16   2       0             0 klogd
Mar 20 01:58:38 cheezum kernel: [ 2691]   101  2691      872        5   2       0             0 dnsmasq
Mar 20 01:58:38 cheezum kernel: [ 2712]     0  2712     1342        6   1     -17         -1000 sshd
Mar 20 01:58:38 cheezum kernel: [ 2771]  1000  2771     3085      467   3     -17         -1000 screen
Mar 20 01:58:38 cheezum kernel: [ 2788]  1000  2788     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2804]  1000  2804     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2820]  1000  2820     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2836]  1000  2836     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2858]  1000  2858     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2887]  1000  2887     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2905]  1000  2905     1451        0   3     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2965]  1000  2965     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 2987]  1000  2987     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3026]  1000  3026     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3054]  1000  3054     1451        0   2     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3109]     0  3109      539        9   3       0             0 cron
Mar 20 01:58:38 cheezum kernel: [ 3146]     1  3146      441       10   2       0             0 uptimed
Mar 20 01:58:38 cheezum kernel: [ 3189]  1000  3189    13778      783   3     -17         -1000 irssi
Mar 20 01:58:38 cheezum kernel: [ 3203]  1001  3203     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3238]  1000  3238     1450        0   1       0             0 bash
Mar 20 01:58:38 cheezum kernel: [ 3453]  1000  3453     1451        0   3     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 3652]     0  3652      786        7   2     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 3656]     0  3656      786        8   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 3658]     0  3658      786        6   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 3696]     0  3696     1816      101   3     -17         -1000 tcpdump
Mar 20 01:58:38 cheezum kernel: [12409]  1001 12409     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [12430]  1001 12430     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15137]  1000 15137     1451        0   2     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15154]  1000 15154      823        0   0     -17         -1000 less
Mar 20 01:58:38 cheezum kernel: [15589]  1000 15589     1451        0   3     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15606]  1000 15606      636      144   2     -17         -1000 htop
Mar 20 01:58:38 cheezum kernel: [15607]  1000 15607     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15623]  1000 15623     1219        0   0     -17         -1000 python
Mar 20 01:58:38 cheezum kernel: [15710]  1000 15710     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15726]     0 15726     2046      561   3     -17         -1000 iotop
Mar 20 01:58:38 cheezum kernel: [15727]  1000 15727     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15743]  1000 15743      473       13   3     -17         -1000 vmstat
Mar 20 01:58:38 cheezum kernel: [15760]  1000 15760     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15776]  1000 15776      516       86   3     -17         -1000 iostat
Mar 20 01:58:38 cheezum kernel: [15825]  1000 15825     1451        0   2     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15841]     0 15841     2619       37   2     -17         -1000 tcpdump
Mar 20 01:58:38 cheezum kernel: [15867]  1000 15867     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [15883]  1000 15883     4229        3   0     -17         -1000 nano
Mar 20 01:58:38 cheezum kernel: [16026]  1000 16026     1451        0   0     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [16050]  1000 16050     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [16214]     0 16214      786        9   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [16660]     0 16660      786        7   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 7028]  1000  7028     1451        1   3     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [14552]  1000 14552     1451        0   2     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [16812]   111 16812     2713        0   1     -17         -1000 nsd
Mar 20 01:58:38 cheezum kernel: [19295]  1000 19295      823        0   3     -17         -1000 less
Mar 20 01:58:38 cheezum kernel: [32143]  1000 32143     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [32190]  1000 32190      823        0   1     -17         -1000 less
Mar 20 01:58:38 cheezum kernel: [ 4050]  1000  4050      786        8   3     -17         -1000 tail
Mar 20 01:58:38 cheezum kernel: [ 1839]  1000  1839     1451        0   1     -17         -1000 bash
Mar 20 01:58:38 cheezum kernel: [ 1873]  1000  1873      823        0   1     -17         -1000 less
Mar 20 01:58:38 cheezum kernel: [31259]   112 31259    34250        0   1     -17         -1000 apt-cacher-ng
Mar 20 01:58:38 cheezum kernel: [13773]   111 13773     1048        0   3     -17         -1000 nsd
Mar 20 01:58:38 cheezum kernel: [13774]   111 13774     1192        0   2     -17         -1000 nsd
Mar 20 01:58:38 cheezum kernel: [31147]  1000 31147      474       60   3       0             0 vmstat
Mar 20 01:58:38 cheezum kernel: [  929]  1000   929     1452        0   2     -17         -1000 bash
Mar 20 01:58:39 cheezum kernel: [  949]  1000   949      824        0   2     -17         -1000 less
Mar 20 01:58:39 cheezum kernel: [ 1021]    33  1021     1826      107   2       0             0 lighttpd
Mar 20 01:58:39 cheezum kernel: [ 1036]    33  1036     1619       68   3       0             0 python
Mar 20 01:58:39 cheezum kernel: [ 1068]    33  1068     1586      120   2       0             0 python
Mar 20 01:58:39 cheezum kernel: [ 1089]    33  1089     1586       74   2       0             0 python
Mar 20 01:58:39 cheezum kernel: [ 1109]    33  1109     1586       92   3       0             0 python
Mar 20 01:58:39 cheezum kernel: [ 1135]    33  1135     1586       78   3       0             0 python
Mar 20 01:58:39 cheezum kernel: [ 3980]  1000  3980     1452        0   0     -17         -1000 bash
Mar 20 01:58:39 cheezum kernel: [ 3998]  1000  3998      972        0   0     -17         -1000 nano
Mar 20 01:58:39 cheezum kernel: [ 4637]  1000  4637     1452        0   3     -17         -1000 bash
Mar 20 01:58:39 cheezum kernel: [ 4654]  1000  4654      972        0   0     -17         -1000 nano
Mar 20 01:58:39 cheezum kernel: Out of memory: Kill process 1021 (lighttpd) score 2 or sacrifice child
Mar 20 01:58:39 cheezum kernel: Killed process 1036 (python) total-vm:6476kB, anon-rss:272kB, file-rss:0kB
Mar 20 02:01:38 cheezum syslogd 1.5.0#1ubuntu1: restart.