RT-AX89X going out of memory every 10 minutes (asd process?)

[shievan]

Hello, my router froze up overnight, so I rebooted it, and ever since the logs are showing that it is going OOM about every 10 minutes. First it starts spamming:

May 17 10:44:26 kernel: [ 5872.720019] wlan: [0:E:QDF] NBUF alloc failed 12107B @ dp_get_ppdu_desc:3430
May 17 10:44:26 kernel: [ 5873.235497] wlan: [0:E:QDF] NBUF alloc failed 12107B @ dp_get_ppdu_desc:3430
May 17 10:44:27 kernel: [ 5873.747543] wlan: [0:E:QDF] NBUF alloc failed 12107B @ dp_get_ppdu_desc:3430
May 17 10:44:27 kernel: [ 5874.259510] wlan: [0:E:QDF] NBUF alloc failed 12107B @ dp_get_ppdu_desc:3430
May 17 10:44:28 kernel: [ 5874.771556] wlan: [0:E:QDF] NBUF alloc failed 12107B @ dp_get_ppdu_desc:3430

Then it looks like a high memory killer kicks in and kills a process called asd:

May 17 10:44:30 kernel: [ 5876.819570] wlan: [0:E:QDF] NBUF alloc failed 12107B @ dp_get_ppdu_desc:3430
May 17 10:44:31 kernel: [ 5877.219109] mcsd invoked oom-killer: gfp_mask=0x24004c0, order=2, oom_score_adj=0
May 17 10:44:31 kernel: [ 5877.219144] CPU: 2 PID: 20987 Comm: mcsd Tainted: P                4.4.60 #1
May 17 10:44:31 kernel: [ 5877.225567] Hardware name: Generic DT based system
May 17 10:44:31 kernel: [ 5877.232709] [<8022001c>] (unwind_backtrace) from [<8021c8c4>] (show_stack+0x10/0x14)
May 17 10:44:31 kernel: [ 5877.237296] [<8021c8c4>] (show_stack) from [<803b8590>] (dump_stack+0x78/0x98)
May 17 10:44:31 kernel: [ 5877.245195] [<803b8590>] (dump_stack) from [<802a7d38>] (dump_header+0x44/0x164)
May 17 10:44:31 kernel: [ 5877.252241] [<802a7d38>] (dump_header) from [<802a825c>] (oom_kill_process+0xcc/0x448)
May 17 10:44:31 kernel: [ 5877.259775] [<802a825c>] (oom_kill_process) from [<802a8924>] (out_of_memory+0x2e4/0x354)
May 17 10:44:31 kernel: [ 5877.267520] [<802a8924>] (out_of_memory) from [<802ac4d4>] (__alloc_pages_nodemask+0x67c/0x738)
May 17 10:44:31 kernel: [ 5877.275752] [<802ac4d4>] (__alloc_pages_nodemask) from [<802ac5a0>] (__get_free_pages+0x10/0x24)
May 17 10:44:31 kernel: [ 5877.284275] [<802ac5a0>] (__get_free_pages) from [<80225d70>] (pgd_alloc+0x18/0x144)
May 17 10:44:31 kernel: [ 5877.293301] [<80225d70>] (pgd_alloc) from [<80228478>] (mm_init+0xcc/0x138)
May 17 10:44:31 kernel: [ 5877.301027] [<80228478>] (mm_init) from [<802e1854>] (do_execveat_common+0x284/0x5f4)
May 17 10:44:31 kernel: [ 5877.307697] [<802e1854>] (do_execveat_common) from [<802e1bf0>] (do_execve+0x2c/0x34)
May 17 10:44:31 kernel: [ 5877.315695] [<802e1bf0>] (do_execve) from [<80209bc0>] (ret_fast_syscall+0x0/0x34)
May 17 10:44:31 kernel: [ 5877.323544] Mem-Info:
May 17 10:44:31 kernel: [ 5877.331211] active_anon:161197 inactive_anon:1485 isolated_anon:0
May 17 10:44:31 kernel: [ 5877.331211]  active_file:90 inactive_file:127 isolated_file:7
May 17 10:44:31 kernel: [ 5877.331211]  unevictable:0 dirty:52 writeback:1 unstable:0
May 17 10:44:31 kernel: [ 5877.331211]  slab_reclaimable:875 slab_unreclaimable:31662
May 17 10:44:31 kernel: [ 5877.331211]  mapped:220 shmem:1513 pagetables:602 bounce:0
May 17 10:44:31 kernel: [ 5877.331211]  free:4778 free_pcp:4 free_cma:0
May 17 10:44:31 kernel: [ 5877.331557] wlan: [0:E:QDF] NBUF alloc failed 12107B @ dp_get_ppdu_desc:3430
May 17 10:44:31 kernel: [ 5877.365795] Normal free:18404kB min:3752kB low:4688kB high:5628kB active_anon:644788kB inactive_anon:5940kB active_file:648kB inactive_file:916kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:898048kB managed:881836kB mlocked:0kB dirty:208kB writeback:4kB mapped:1176kB shmem:6052kB slab_reclaimable:3500kB slab_unreclaimable:126648kB kernel_stack:1432kB pagetables:2408kB unstable:0kB bounce:0kB free_pcp:544kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_
May 17 10:44:31 kernel: [ 5877.395533] lowmem_reserve[]: 0 0 0
May 17 10:44:31 kernel: [ 5877.421214] Normal: 4597*4kB (U) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 18388kB
May 17 10:44:31 kernel: [ 5877.431371] 2010 total pagecache pages
May 17 10:44:31 kernel: [ 5877.432161] 0 pages in swap cache
May 17 10:44:31 kernel: [ 5877.435905] Swap cache stats: add 0, delete 0, find 0/0
May 17 10:44:31 kernel: [ 5877.439280] Free swap  = 0kB
May 17 10:44:31 kernel: [ 5877.444308] Total swap = 0kB
May 17 10:44:31 kernel: [ 5877.447454] 224512 pages RAM
May 17 10:44:31 kernel: [ 5877.450298] 0 pages HighMem/MovableOnly
May 17 10:44:31 kernel: [ 5877.453163] 4053 pages reserved
May 17 10:44:31 kernel: [ 5877.457046] Out of memory: Kill process 9710 (asd) score 684 or sacrifice child
May 17 10:44:31 kernel: [ 5877.459867] Killed process 9710 (asd) total-vm:625440kB, anon-rss:621348kB, file-rss:60kB

 Looking back at when the router locked up initially overnight, it seems like it tried to look for a firmware update automatically around 4am, (which is strange because I have auto updates turned off), then started spamming the 'NBUF alloc failed' message:

May 17 03:48:01 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7616)]do webs_update
May 17 03:48:05 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7634)]retrieve firmware information
May 17 03:48:05 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7649)]fimrware update check first time
May 17 03:48:05 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7682)]no need to upgrade firmware
May 17 03:48:35 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7616)]do webs_update
May 17 03:48:35 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7634)]retrieve firmware information
May 17 03:48:35 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7654)]fimrware update check once
May 17 03:49:05 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7616)]do webs_update
May 17 03:49:05 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7634)]retrieve firmware information
May 17 03:49:05 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7654)]fimrware update check once
May 17 03:49:35 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7616)]do webs_update
May 17 03:49:35 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7634)]retrieve firmware information
May 17 03:49:35 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7649)]fimrware update check first time
May 17 03:49:35 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7682)]no need to upgrade firmware
May 17 03:57:55 kernel: [1382843.922889] wlan: [0:E:QDF] NBUF alloc failed 12107B @ dp_get_ppdu_desc:3430
May 17 03:57:55 kernel: [1382844.435933] wlan: [0:E:QDF] NBUF alloc failed 12107B @ dp_get_ppdu_desc:3430
May 17 03:57:56 kernel: [1382845.012988] wlan: [0:E:QDF] NBUF alloc failed 12107B @ dp_get_ppdu_desc:3430
May 17 03:57:56 kernel: [1382845.525014] wlan: [0:E:QDF] NBUF alloc failed 12107B @ dp_get_ppdu_desc:3430

 Any help greatly appreciated!

[RogSuxAss]

I found this solution on another forum, I was surprised that I had to download the firmware from somewhere else even though I was expecting it to be available on the official ASUS website. 

Another thing I want to ask about is that there was obviously some kind of ''memory leak'' - when the memory became full, it crashed, repeat. So my question is, I suspect that such a thing can shorten the router's lifespan, why is it not being addressed? Usually, the router's memory is consistently half-full, load 5%. However, in the past 24 hours, it has been working harder than ever before. If something happens to it shortly after the warranty expires, I will definitely attribute it to this situation.

[tmaycsu]

I rolled back to the previous FW version and did a factory reset and it fixed the issue for me (at least for the last 10 hours). Hopefully ASUS has a new FW for us soon that’s not a beta to fix the issue for real. 

[tmaycsu]

Another question I have…why hasn’t ASUS pulled the troublesome FW version from being an available update? It’s been a problem for 3 days now and their overall response has been very slow. 

[Jasonfi]

Job security. 🤣

[rain111]

I have installed the beta version that was shared here, hopefully, it will fix this issue temp until ASUS release an update with the fix

[rkalinka]

UPDATE - after a "crash" yesterday (4 hours in), the router has been stable for the past 20 hours or so.  The problem is that the asd.log is showing that "update failed" so I am not sure if ASD is running correctly.  When I opened a ticket with Asus, the indicated that I should factory reset the router and MANUALLY rebuild it (no file import) - which is not something I am willing to do just yet.  When I went to double check my Asus support ticket, the ASUS site shows this now so not sure what I am going to do long term:

####################################

Interruption in Router Product Connectivity and Urgent Mitigation Measures

Dear Valued Customer,
During routine security maintenance, our technical team discovered an error in the configuration of our server settings file, which could potentially cause an interruption in network connectivity on part of the routers.
• Our technical team has urgently addressed the server issue and impacted routers should return to normal operation. If your device was affected, we recommend the following:
1. Manually reboot your router
2. If rebooting does not resolve the issue, please save the settings file, perform a hard reset (factory default), and then re-upload the settings file (follow the directions in the https://www.asus.com/support/FAQ/1050464)
3. If you cannot access the user interface to save settings or perform a reset, you can press the RESET button for about 5-10 seconds until the power LED indicator on the router starts to blink, which means the reset is completed.
https://www.asus.com/support/FAQ/1000925/#m2
If there are any further developments around this issue, we will immediately update our users.
We deeply apologize for any inconvenience this incident may have caused and are committed to preventing such an incident from happening again.
For any further inquiries about your ASUS router, please contact our customer service for support.

USA & Canada Hotline:1-812-282-2787
Support site ASUS ROG

Thank you for your understanding and thank you for choosing ASUS.

[wilson89]

After hard factory reset, it worked for one day. And 24 hours later, it's breaking internet and even the connection to the router, every time I'm on zoom/google meet calls. Well, effing solve it for non-tech people quickly. I don't have time or expertise to fix their problems.

Too bad this is worldwide and they are so bad at it. ASUS routers knocked offline worldwide by bad security update (bleepingcomputer.com)

[MisterChow19]

What should your RAM and CPU be for the AX89X? 

I have a HD attached to mine and I'm seeing 53% RAM usage and ~7% CPU. I assume CPU is only really affected by bandwidth heavy tasks? 

[rain111]

I have no HDD plugged into my router, aside from that, my CPU usage should be hovering around 5~10% and RAM around 50~60%, this is with a total of 20+ connections throughout the house.

Looking at your % usage, it looks about right.

[SJC09]

I have two RP-AC55's that I use as media bridges (these are very simple, low memory, 1 cpu devices) -- one connects to my Arlo camera base station, and I have another that connects to a Tivo Mini that doesn't have WiFi... Both of these devices were the first to freeze up on me.. I also have a 3 node ZenWiFI AX Mini system in AP Mode, that is a bit higher powered - but still not immune to this, one of the nodes went down a few times last week and until I found this thread, I was chasing everything down to see why..   There have been no firmware updates for those devices since last year -- and I don't have automatic updates enabled.  YET, despite this -- these devices are contacting Asus constantly and downloading files apparently without my knowledge, that can bring my network down.  Is there disclosure of this anywhere?   How can we block these downloads from taking place?  Has anyone traced to find the host that the Asus devices download these asd updates from?  I'll just block it at my router level (which thankfully runs OPNSense and not any type of Asus device)

Thanks