RT-AX89X going out of memory every 10 minutes (asd process?)

[shievan]

Hello, my router froze up overnight, so I rebooted it, and ever since the logs are showing that it is going OOM about every 10 minutes. First it starts spamming:

May 17 10:44:26 kernel: [ 5872.720019] wlan: [0:E:QDF] NBUF alloc failed 12107B @ dp_get_ppdu_desc:3430
May 17 10:44:26 kernel: [ 5873.235497] wlan: [0:E:QDF] NBUF alloc failed 12107B @ dp_get_ppdu_desc:3430
May 17 10:44:27 kernel: [ 5873.747543] wlan: [0:E:QDF] NBUF alloc failed 12107B @ dp_get_ppdu_desc:3430
May 17 10:44:27 kernel: [ 5874.259510] wlan: [0:E:QDF] NBUF alloc failed 12107B @ dp_get_ppdu_desc:3430
May 17 10:44:28 kernel: [ 5874.771556] wlan: [0:E:QDF] NBUF alloc failed 12107B @ dp_get_ppdu_desc:3430

Then it looks like a high memory killer kicks in and kills a process called asd:

May 17 10:44:30 kernel: [ 5876.819570] wlan: [0:E:QDF] NBUF alloc failed 12107B @ dp_get_ppdu_desc:3430
May 17 10:44:31 kernel: [ 5877.219109] mcsd invoked oom-killer: gfp_mask=0x24004c0, order=2, oom_score_adj=0
May 17 10:44:31 kernel: [ 5877.219144] CPU: 2 PID: 20987 Comm: mcsd Tainted: P                4.4.60 #1
May 17 10:44:31 kernel: [ 5877.225567] Hardware name: Generic DT based system
May 17 10:44:31 kernel: [ 5877.232709] [<8022001c>] (unwind_backtrace) from [<8021c8c4>] (show_stack+0x10/0x14)
May 17 10:44:31 kernel: [ 5877.237296] [<8021c8c4>] (show_stack) from [<803b8590>] (dump_stack+0x78/0x98)
May 17 10:44:31 kernel: [ 5877.245195] [<803b8590>] (dump_stack) from [<802a7d38>] (dump_header+0x44/0x164)
May 17 10:44:31 kernel: [ 5877.252241] [<802a7d38>] (dump_header) from [<802a825c>] (oom_kill_process+0xcc/0x448)
May 17 10:44:31 kernel: [ 5877.259775] [<802a825c>] (oom_kill_process) from [<802a8924>] (out_of_memory+0x2e4/0x354)
May 17 10:44:31 kernel: [ 5877.267520] [<802a8924>] (out_of_memory) from [<802ac4d4>] (__alloc_pages_nodemask+0x67c/0x738)
May 17 10:44:31 kernel: [ 5877.275752] [<802ac4d4>] (__alloc_pages_nodemask) from [<802ac5a0>] (__get_free_pages+0x10/0x24)
May 17 10:44:31 kernel: [ 5877.284275] [<802ac5a0>] (__get_free_pages) from [<80225d70>] (pgd_alloc+0x18/0x144)
May 17 10:44:31 kernel: [ 5877.293301] [<80225d70>] (pgd_alloc) from [<80228478>] (mm_init+0xcc/0x138)
May 17 10:44:31 kernel: [ 5877.301027] [<80228478>] (mm_init) from [<802e1854>] (do_execveat_common+0x284/0x5f4)
May 17 10:44:31 kernel: [ 5877.307697] [<802e1854>] (do_execveat_common) from [<802e1bf0>] (do_execve+0x2c/0x34)
May 17 10:44:31 kernel: [ 5877.315695] [<802e1bf0>] (do_execve) from [<80209bc0>] (ret_fast_syscall+0x0/0x34)
May 17 10:44:31 kernel: [ 5877.323544] Mem-Info:
May 17 10:44:31 kernel: [ 5877.331211] active_anon:161197 inactive_anon:1485 isolated_anon:0
May 17 10:44:31 kernel: [ 5877.331211]  active_file:90 inactive_file:127 isolated_file:7
May 17 10:44:31 kernel: [ 5877.331211]  unevictable:0 dirty:52 writeback:1 unstable:0
May 17 10:44:31 kernel: [ 5877.331211]  slab_reclaimable:875 slab_unreclaimable:31662
May 17 10:44:31 kernel: [ 5877.331211]  mapped:220 shmem:1513 pagetables:602 bounce:0
May 17 10:44:31 kernel: [ 5877.331211]  free:4778 free_pcp:4 free_cma:0
May 17 10:44:31 kernel: [ 5877.331557] wlan: [0:E:QDF] NBUF alloc failed 12107B @ dp_get_ppdu_desc:3430
May 17 10:44:31 kernel: [ 5877.365795] Normal free:18404kB min:3752kB low:4688kB high:5628kB active_anon:644788kB inactive_anon:5940kB active_file:648kB inactive_file:916kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:898048kB managed:881836kB mlocked:0kB dirty:208kB writeback:4kB mapped:1176kB shmem:6052kB slab_reclaimable:3500kB slab_unreclaimable:126648kB kernel_stack:1432kB pagetables:2408kB unstable:0kB bounce:0kB free_pcp:544kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_
May 17 10:44:31 kernel: [ 5877.395533] lowmem_reserve[]: 0 0 0
May 17 10:44:31 kernel: [ 5877.421214] Normal: 4597*4kB (U) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 18388kB
May 17 10:44:31 kernel: [ 5877.431371] 2010 total pagecache pages
May 17 10:44:31 kernel: [ 5877.432161] 0 pages in swap cache
May 17 10:44:31 kernel: [ 5877.435905] Swap cache stats: add 0, delete 0, find 0/0
May 17 10:44:31 kernel: [ 5877.439280] Free swap  = 0kB
May 17 10:44:31 kernel: [ 5877.444308] Total swap = 0kB
May 17 10:44:31 kernel: [ 5877.447454] 224512 pages RAM
May 17 10:44:31 kernel: [ 5877.450298] 0 pages HighMem/MovableOnly
May 17 10:44:31 kernel: [ 5877.453163] 4053 pages reserved
May 17 10:44:31 kernel: [ 5877.457046] Out of memory: Kill process 9710 (asd) score 684 or sacrifice child
May 17 10:44:31 kernel: [ 5877.459867] Killed process 9710 (asd) total-vm:625440kB, anon-rss:621348kB, file-rss:60kB

 Looking back at when the router locked up initially overnight, it seems like it tried to look for a firmware update automatically around 4am, (which is strange because I have auto updates turned off), then started spamming the 'NBUF alloc failed' message:

May 17 03:48:01 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7616)]do webs_update
May 17 03:48:05 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7634)]retrieve firmware information
May 17 03:48:05 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7649)]fimrware update check first time
May 17 03:48:05 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7682)]no need to upgrade firmware
May 17 03:48:35 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7616)]do webs_update
May 17 03:48:35 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7634)]retrieve firmware information
May 17 03:48:35 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7654)]fimrware update check once
May 17 03:49:05 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7616)]do webs_update
May 17 03:49:05 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7634)]retrieve firmware information
May 17 03:49:05 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7654)]fimrware update check once
May 17 03:49:35 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7616)]do webs_update
May 17 03:49:35 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7634)]retrieve firmware information
May 17 03:49:35 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7649)]fimrware update check first time
May 17 03:49:35 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7682)]no need to upgrade firmware
May 17 03:57:55 kernel: [1382843.922889] wlan: [0:E:QDF] NBUF alloc failed 12107B @ dp_get_ppdu_desc:3430
May 17 03:57:55 kernel: [1382844.435933] wlan: [0:E:QDF] NBUF alloc failed 12107B @ dp_get_ppdu_desc:3430
May 17 03:57:56 kernel: [1382845.012988] wlan: [0:E:QDF] NBUF alloc failed 12107B @ dp_get_ppdu_desc:3430
May 17 03:57:56 kernel: [1382845.525014] wlan: [0:E:QDF] NBUF alloc failed 12107B @ dp_get_ppdu_desc:3430

 Any help greatly appreciated!

[chrissrogers]

Also experiencing this issue. My logs show

NBUF alloc failed 12107B @ dp_get_ppdu_desc:3430

prior to OOMing and falling over. Began overnight. I have not performed a firmware downgrade yet. Automatic firmware updates are disabled on my router, and were disabled overnight.

[AllYourBass]

I manually downgraded the firmware to 3.0.0.4.386_47191-g9ddcc31, and can confirm that the wifi does stay up longer than 10 minutes (but I don't know how long it will last). The log file is littered with out of memory OOM entries that start 10 minutes after the first FAUPGRADE log message.

I also contacted tech support and they escalated the ticket and I'm uploading log files now from both firmware versions.

[shievan]

I wonder if the firmware downgrade really helps or is a red herring. I'm still on the same firmware I started with (3.0.0.4.386_47468). I just restarted a couple times, and my router is now "working", but with the OOM log spam I mentioned every 10ish minutes. 

[AllYourBass]

I'll take a red herring that keeps the wifi up without me having to reboot the router every 10 minutes. The kids are coming home from school soon, and I couldn't handle that pressure!

Sent off my logs to Asus just now. Hopefully it helps them figure out the issue and resolve it.

[e1337ninja]

My router is doing the exact same thing. Started last night around 4am Central Time.

[cphedrick]

Same problem here.  My RT-AX89X disconnects every 10 minutes or so.  It began overnight sometime ad checking the logs shows the same NBUF allocation failures as OP.  FWIW, I do have automatic updates active, and have the latest firmware version installed.

[doe152]

Having the same issue here since sometime during the night.  Factory reset three times.  Still after a few minutes the WiFi is unreachable and cannot get to the router's web interface page.

[breadswonders]

I am also having this exact issue which started this morning. From the logs, I can see two things potentially related to the failure here: 

• The Inadyn DDNS client
• The firmware update check service/process

Here are the relevant Inadyn logs:

May 17 07:35:50 watchdog: start ddns.
May 17 07:35:50 rc_service: watchdog 1487:notify_rc restart_ddns
May 17 07:35:50 start_ddns: update WWW.ASUS.COM update@asus.com, wan_unit 0
May 17 07:35:50 start_ddns: Clear ddns cache.
May 17 07:35:50 start_ddns: Start Inadyn.
May 17 07:35:50 inadyn[3490]: In-a-dyn version 2.8.1 -- Dynamic DNS update client.
May 17 07:35:50 inadyn[3490]: Failed resolving hostname <REDACTED>.asuscomm.com: Name or service not known
May 17 07:35:50 inadyn[3490]: Update forced for alias <REDACTED>.asuscomm.com, new IP# 192.168.1.66
May 17 07:35:51 APSTATS: band 0: beacon success: 195/0/0/0 state: ENABLED
May 17 07:35:51 inadyn[3490]: alias address=<192.168.1.66>
May 17 07:35:51 inadyn[3490]: request<GET /ddns/update.jsp?hostname=<REDACTED>.asuscomm.com&myip=192.168.1.66&model=RT-AX89U&fw_ver=3.0.0.4.386_47468-g73fe1fe HTTP/1.0^M Authorization: Basic <REDACTED>^M Host: ns1.asuscomm.com^M User-Agent: inadyn/2.8.1 https://github.com/troglobit/inadyn/issues^M ^M >
...
May 17 07:36:06 inadyn[3490]: [response_update]HTTP/1.1 299 |Invalid IP format| 192.168.1.66^M Date: Wed, 17 May 2023 13:35:51 GMT^M Server: Apache^M Content-Length: 0^M Content-Type: text/html; charset=UTF-8^M ^M
May 17 07:36:06 inadyn[3490]: Invalid IP address
May 17 07:36:06 inadyn[3490]: Fatal error in DDNS server response:
May 17 07:36:06 inadyn[3490]: [299 |Invalid IP format| 192.168.1.66] 

Firmware update check:

May 17 07:37:20 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7577)]periodic_check AM 4:49
May 17 07:37:20 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7616)]do webs_update
May 17 07:37:25 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7634)]retrieve firmware information
May 17 07:37:25 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7649)]fimrware update check first time
May 17 07:37:25 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7682)]no need to upgrade firmware

I then get the following until the router attempts to clear the memory, or crashes:

May 17 09:25:23 kernel: [  642.027787] wlan: [0:E:QDF] NBUF alloc failed 12107B @ dp_get_ppdu_desc:3430

Perhaps this entire issue is just being caused by an unexpected HTTP response just not being handled correctly?

[MorsCanis]

Manually updated back to latest firmware. No long locking up. Logs show that it's not checking for updates every 10 mins. Pretty sure they're server messed up and they took it down to fix it. 

Auto updates are turned off, but it was still checking anyways.

[doe152]

I just did the same.  Been trouble-free for longer than I have been all day between the factory resets and so on.  Fingers crossed. 

Downgraded to v3.0.0.4.386.47191.