X11 crashes on Pi 4 with Trixie and a viewer connected to RealVNC server

[Mastiffen]

Describe the bug

I use RealVNC's server vncserver-x11-serviced (of reasons that isn't relevant to this). I recently installed the latest Trixie via Raspberry Pi Imager, and when it is connected for a while, V3D crashes. It can be triggered by using the command line interface, in my case a DD image copy, which I use to back up all my Pi's. But it happens also when the viewer is connected for a long time. This does not happen when I do the same on a Pi 5, from what I have seen so far. I have also tried several times to do this without any viewer connected to the VNC server, and it does not happen in that case.

Steps to reproduce the behaviour

1. Change from Wayland to X11 in raspi-config
2. Activate the VNC server with sudo systemctl enable vncserver-x11-serviced
3. Start an image copy, like sudo dd if=/dev/mmcblk0 of=/media/pi/Backups/Trixie-Pi4-test-20260713.img bs=512 status=progress (in my case the directory Backups is a Samba share on my Windows server either at home or at my cabin, the same thing happens in both places.
4. Wait. It will crash usually on the first run, at least on the second every time I have tried so far.

Device (s)

Raspberry Pi CM4

System

https://pastebin.com/Tk2R6ftx

Logs

One time:
v3d fec00000.v3d: MMU error from client L2T (2) at 0x0, pte invalid

Another time:

[ 1648.576308] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 1648.576361] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
[ 1648.576487] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 1648.576511] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
[ 1649.600473] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 1649.600557] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
[ 1650.624579] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 1650.624631] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
[ 1651.648726] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 1651.648782] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
[ 1652.672852] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 1652.672907] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
[ 1653.696997] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 1653.697050] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
[ 1654.721261] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 1654.721313] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
[ 1655.745273] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 1655.745324] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
[ 1656.769412] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 1656.769466] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
[ 1657.793558] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 1657.793621] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
[ 1658.817694] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 1658.817745] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
[ 1659.841837] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 1659.841892] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
[ 1660.866003] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 1660.866057] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
[ 1661.894123] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 1661.894180] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
[ 1662.914274] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 1662.914335] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
[ 1663.938240] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 1663.938295] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
[ 1664.959658] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 1664.959716] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
[ 1665.977924] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 1665.978005] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000

Additional context

While I don't know if this is a bug in RealVNC or the kernel, it does not ever happen in Bookworm, and it has not happened in any earlier version that I have used for the previous ten years. And all my Bookworm Pi's are updated, so I assume they are running the same version of RealVNC server as my so far two Trixie Pi 4's.

[Mastiffen]

Her are two different screenshots, after the same thing has happened. The first with dd and the second with a VNC Viewer connected for a long time:

[popcornmix]

Can you confirm kernel version (output of uname -a).
The MMU error messages are typically benign.
The Resetting GPU for hang are likely an issue - do you always see these messages following a "crash"

You say "Wait. It will crash usually on the first run, at least on the second every time I have tried so far."
Describe exactly what the crash is? Does vnc just freeze? Does ssh session keep running?
Can you access the machine directly (e.g. through mouse/keyboard).

[Mastiffen]

uname -a
Linux Hytte-Pi 6.18.34+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.18.34-1+rpt1 (2026-06-09) aarch64 GNU/Linux

I don't see the messages every time, but they seem to be there most of the time.

The crash varies a bit. Sometimes I can move the mouse, but stuff gets black in rows below it, other times the screen locks up completely. It is same on a monitor, as long as the VNC Viewer has been connected to get the crash X11 has the same problem on both VNC and monitor.

SSH works as it should, I can reboot it from there.

[popcornmix]

run

cat /sys/devices/platform/v3dbus/fec00000.v3d/power/control

You should see auto. Run:

echo on | sudo tee /sys/devices/platform/v3dbus/fec00000.v3d/power/control

That should disable suspend of v3d driver. Does that help?

[Mastiffen]

"auto" it was, yeah.

I have run the command, the result was "on" and I am setting the dd task to start again now.

I was just about to post that I let the Pi run for longer after it had crashed, and it started putting out some new stuff in dmesg:

[ 7375.043841] v3d fec00000.v3d: MMU error from client L2T (3) at 0x0, pte invalid
[ 7376.060075] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 7376.060132] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
[ 7377.084073] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 7377.084123] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
[ 7378.108086] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 7378.108138] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
[ 7379.132102] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 7379.132154] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
[ 7379.644098] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 7379.644152] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
[ 7380.156166] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 7380.156245] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
[ 7380.668313] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 7380.668433] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
[ 7381.180218] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 7381.180353] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
[ 7381.692296] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 7381.692418] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
[ 7382.204619] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 7382.204709] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
[ 7382.716287] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 7382.716375] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
[ 7383.740153] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 7383.740204] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
[ 7384.800164] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 7384.800219] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
[ 7385.820723] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 7385.820773] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
[ 7386.332191] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 7386.332240] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
[ 7387.356209] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 7387.356256] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
[ 7388.412380] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 7388.412432] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
[ 7389.436228] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 7389.436277] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
[ 7389.436385] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 7389.436411] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
[ 7390.460234] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* Resetting GPU for hang.
[ 7390.460293] v3d fec00000.v3d: [drm:v3d_reset [v3d]] *ERROR* V3D_ERR_STAT: 0x00001000
[ 7613.503165] INFO: task kworker/2:0:62785 blocked for more than 120 seconds.
[ 7613.503196]       Tainted: G         C          6.18.34+rpt-rpi-v8 #1 Debian 1:6.18.34-1+rpt1
[ 7613.503206] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 7613.503213] task:kworker/2:0     state:D stack:0     pid:62785 tgid:62785 ppid:2      task_flags:0x4208060 flags:0x00000010
[ 7613.503237] Workqueue: events drm_sched_job_timedout [gpu_sched]
[ 7613.503284] Call trace:
[ 7613.503292]  __switch_to+0xf0/0x1c8 (T)
[ 7613.503314]  __schedule+0x3c8/0xc18
[ 7613.503327]  schedule+0x3c/0x120
[ 7613.503339]  schedule_timeout+0xdc/0x130
[ 7613.503356]  dma_fence_default_wait+0x1f4/0x238
[ 7613.503371]  dma_fence_wait_timeout+0xc4/0x220
[ 7613.503384]  drm_sched_stop+0x13c/0x220 [gpu_sched]
[ 7613.503415]  v3d_gpu_reset_for_timeout.isra.0+0x68/0x120 [v3d]
[ 7613.503460]  v3d_bin_job_timedout+0x80/0xa0 [v3d]
[ 7613.503499]  drm_sched_job_timedout+0x80/0x180 [gpu_sched]
[ 7613.503527]  process_one_work+0x158/0x3b8
[ 7613.503547]  worker_thread+0x19c/0x328
[ 7613.503562]  kthread+0x14c/0x210
[ 7613.503577]  ret_from_fork+0x10/0x20
[ 7734.336699] INFO: task kworker/2:0:62785 blocked for more than 241 seconds.
[ 7734.336736]       Tainted: G         C          6.18.34+rpt-rpi-v8 #1 Debian 1:6.18.34-1+rpt1
[ 7734.336748] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 7734.336756] task:kworker/2:0     state:D stack:0     pid:62785 tgid:62785 ppid:2      task_flags:0x4208060 flags:0x00000010
[ 7734.336785] Workqueue: events drm_sched_job_timedout [gpu_sched]
[ 7734.336840] Call trace:
[ 7734.336850]  __switch_to+0xf0/0x1c8 (T)
[ 7734.336875]  __schedule+0x3c8/0xc18
[ 7734.336890]  schedule+0x3c/0x120
[ 7734.336904]  schedule_timeout+0xdc/0x130
[ 7734.336922]  dma_fence_default_wait+0x1f4/0x238
[ 7734.336940]  dma_fence_wait_timeout+0xc4/0x220
[ 7734.336956]  drm_sched_stop+0x13c/0x220 [gpu_sched]
[ 7734.336991]  v3d_gpu_reset_for_timeout.isra.0+0x68/0x120 [v3d]
[ 7734.337043]  v3d_bin_job_timedout+0x80/0xa0 [v3d]
[ 7734.337087]  drm_sched_job_timedout+0x80/0x180 [gpu_sched]
[ 7734.337120]  process_one_work+0x158/0x3b8
[ 7734.337142]  worker_thread+0x19c/0x328
[ 7734.337161]  kthread+0x14c/0x210
[ 7734.337178]  ret_from_fork+0x10/0x20
[ 7855.174124] INFO: task kworker/2:0:62785 blocked for more than 362 seconds.
[ 7855.174147]       Tainted: G         C          6.18.34+rpt-rpi-v8 #1 Debian 1:6.18.34-1+rpt1
[ 7855.174152] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 7855.174156] task:kworker/2:0     state:D stack:0     pid:62785 tgid:62785 ppid:2      task_flags:0x4208060 flags:0x00000010
[ 7855.174170] Workqueue: events drm_sched_job_timedout [gpu_sched]
[ 7855.174197] Call trace:
[ 7855.174201]  __switch_to+0xf0/0x1c8 (T)
[ 7855.174213]  __schedule+0x3c8/0xc18
[ 7855.174219]  schedule+0x3c/0x120
[ 7855.174225]  schedule_timeout+0xdc/0x130
[ 7855.174233]  dma_fence_default_wait+0x1f4/0x238
[ 7855.174241]  dma_fence_wait_timeout+0xc4/0x220
[ 7855.174247]  drm_sched_stop+0x13c/0x220 [gpu_sched]
[ 7855.174262]  v3d_gpu_reset_for_timeout.isra.0+0x68/0x120 [v3d]
[ 7855.174286]  v3d_bin_job_timedout+0x80/0xa0 [v3d]
[ 7855.174304]  drm_sched_job_timedout+0x80/0x180 [gpu_sched]
[ 7855.174317]  process_one_work+0x158/0x3b8
[ 7855.174327]  worker_thread+0x19c/0x328
[ 7855.174334]  kthread+0x14c/0x210
[ 7855.174341]  ret_from_fork+0x10/0x20
[ 7976.003690] INFO: task kworker/2:0:62785 blocked for more than 483 seconds.
[ 7976.003711]       Tainted: G         C          6.18.34+rpt-rpi-v8 #1 Debian 1:6.18.34-1+rpt1
[ 7976.003717] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 7976.003721] task:kworker/2:0     state:D stack:0     pid:62785 tgid:62785 ppid:2      task_flags:0x4208060 flags:0x00000010
[ 7976.003734] Workqueue: events drm_sched_job_timedout [gpu_sched]
[ 7976.003761] Call trace:
[ 7976.003766]  __switch_to+0xf0/0x1c8 (T)
[ 7976.003778]  __schedule+0x3c8/0xc18
[ 7976.003784]  schedule+0x3c/0x120
[ 7976.003789]  schedule_timeout+0xdc/0x130
[ 7976.003798]  dma_fence_default_wait+0x1f4/0x238
[ 7976.003806]  dma_fence_wait_timeout+0xc4/0x220
[ 7976.003812]  drm_sched_stop+0x13c/0x220 [gpu_sched]
[ 7976.003827]  v3d_gpu_reset_for_timeout.isra.0+0x68/0x120 [v3d]
[ 7976.003850]  v3d_bin_job_timedout+0x80/0xa0 [v3d]
[ 7976.003868]  drm_sched_job_timedout+0x80/0x180 [gpu_sched]
[ 7976.003881]  process_one_work+0x158/0x3b8
[ 7976.003890]  worker_thread+0x19c/0x328
[ 7976.003898]  kthread+0x14c/0x210
[ 7976.003905]  ret_from_fork+0x10/0x20
[ 8096.837190] INFO: task kworker/2:0:62785 blocked for more than 604 seconds.
[ 8096.837212]       Tainted: G         C          6.18.34+rpt-rpi-v8 #1 Debian 1:6.18.34-1+rpt1
[ 8096.837218] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 8096.837222] task:kworker/2:0     state:D stack:0     pid:62785 tgid:62785 ppid:2      task_flags:0x4208060 flags:0x00000010
[ 8096.837235] Workqueue: events drm_sched_job_timedout [gpu_sched]
[ 8096.837262] Call trace:
[ 8096.837266]  __switch_to+0xf0/0x1c8 (T)
[ 8096.837278]  __schedule+0x3c8/0xc18
[ 8096.837284]  schedule+0x3c/0x120
[ 8096.837290]  schedule_timeout+0xdc/0x130
[ 8096.837298]  dma_fence_default_wait+0x1f4/0x238
[ 8096.837306]  dma_fence_wait_timeout+0xc4/0x220
[ 8096.837312]  drm_sched_stop+0x13c/0x220 [gpu_sched]
[ 8096.837327]  v3d_gpu_reset_for_timeout.isra.0+0x68/0x120 [v3d]
[ 8096.837350]  v3d_bin_job_timedout+0x80/0xa0 [v3d]
[ 8096.837368]  drm_sched_job_timedout+0x80/0x180 [gpu_sched]
[ 8096.837381]  process_one_work+0x158/0x3b8
[ 8096.837391]  worker_thread+0x19c/0x328
[ 8096.837398]  kthread+0x14c/0x210
[ 8096.837405]  ret_from_fork+0x10/0x20
[ 8217.670657] INFO: task kworker/2:0:62785 blocked for more than 724 seconds.
[ 8217.670678]       Tainted: G         C          6.18.34+rpt-rpi-v8 #1 Debian 1:6.18.34-1+rpt1
[ 8217.670684] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 8217.670687] task:kworker/2:0     state:D stack:0     pid:62785 tgid:62785 ppid:2      task_flags:0x4208060 flags:0x00000010
[ 8217.670700] Workqueue: events drm_sched_job_timedout [gpu_sched]
[ 8217.670728] Call trace:
[ 8217.670732]  __switch_to+0xf0/0x1c8 (T)
[ 8217.670744]  __schedule+0x3c8/0xc18
[ 8217.670750]  schedule+0x3c/0x120
[ 8217.670755]  schedule_timeout+0xdc/0x130
[ 8217.670763]  dma_fence_default_wait+0x1f4/0x238
[ 8217.670770]  dma_fence_wait_timeout+0xc4/0x220
[ 8217.670777]  drm_sched_stop+0x13c/0x220 [gpu_sched]
[ 8217.670791]  v3d_gpu_reset_for_timeout.isra.0+0x68/0x120 [v3d]
[ 8217.670814]  v3d_bin_job_timedout+0x80/0xa0 [v3d]
[ 8217.670832]  drm_sched_job_timedout+0x80/0x180 [gpu_sched]
[ 8217.670845]  process_one_work+0x158/0x3b8
[ 8217.670854]  worker_thread+0x19c/0x328
[ 8217.670862]  kthread+0x14c/0x210
[ 8217.670869]  ret_from_fork+0x10/0x20
[ 8338.503881] INFO: task kworker/2:0:62785 blocked for more than 845 seconds.
[ 8338.503914]       Tainted: G         C          6.18.34+rpt-rpi-v8 #1 Debian 1:6.18.34-1+rpt1
[ 8338.503925] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 8338.503932] task:kworker/2:0     state:D stack:0     pid:62785 tgid:62785 ppid:2      task_flags:0x4208060 flags:0x00000010
[ 8338.503957] Workqueue: events drm_sched_job_timedout [gpu_sched]
[ 8338.504005] Call trace:
[ 8338.504013]  __switch_to+0xf0/0x1c8 (T)
[ 8338.504035]  __schedule+0x3c8/0xc18
[ 8338.504047]  schedule+0x3c/0x120
[ 8338.504059]  schedule_timeout+0xdc/0x130
[ 8338.504075]  dma_fence_default_wait+0x1f4/0x238
[ 8338.504091]  dma_fence_wait_timeout+0xc4/0x220
[ 8338.504105]  drm_sched_stop+0x13c/0x220 [gpu_sched]
[ 8338.504136]  v3d_gpu_reset_for_timeout.isra.0+0x68/0x120 [v3d]
[ 8338.504182]  v3d_bin_job_timedout+0x80/0xa0 [v3d]
[ 8338.504221]  drm_sched_job_timedout+0x80/0x180 [gpu_sched]
[ 8338.504249]  process_one_work+0x158/0x3b8
[ 8338.504269]  worker_thread+0x19c/0x328
[ 8338.504285]  kthread+0x14c/0x210
[ 8338.504300]  ret_from_fork+0x10/0x20
[ 8459.337128] INFO: task kworker/2:0:62785 blocked for more than 966 seconds.
[ 8459.337165]       Tainted: G         C          6.18.34+rpt-rpi-v8 #1 Debian 1:6.18.34-1+rpt1
[ 8459.337177] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 8459.337185] task:kworker/2:0     state:D stack:0     pid:62785 tgid:62785 ppid:2      task_flags:0x4208060 flags:0x00000010
[ 8459.337214] Workqueue: events drm_sched_job_timedout [gpu_sched]
[ 8459.337270] Call trace:
[ 8459.337278]  __switch_to+0xf0/0x1c8 (T)
[ 8459.337303]  __schedule+0x3c8/0xc18
[ 8459.337317]  schedule+0x3c/0x120
[ 8459.337331]  schedule_timeout+0xdc/0x130
[ 8459.337350]  dma_fence_default_wait+0x1f4/0x238
[ 8459.337368]  dma_fence_wait_timeout+0xc4/0x220
[ 8459.337384]  drm_sched_stop+0x13c/0x220 [gpu_sched]
[ 8459.337420]  v3d_gpu_reset_for_timeout.isra.0+0x68/0x120 [v3d]
[ 8459.337472]  v3d_bin_job_timedout+0x80/0xa0 [v3d]
[ 8459.337517]  drm_sched_job_timedout+0x80/0x180 [gpu_sched]
[ 8459.337551]  process_one_work+0x158/0x3b8
[ 8459.337573]  worker_thread+0x19c/0x328
[ 8459.337592]  kthread+0x14c/0x210
[ 8459.337609]  ret_from_fork+0x10/0x20
[ 8580.170305] INFO: task kworker/2:0:62785 blocked for more than 1087 seconds.
[ 8580.170327]       Tainted: G         C          6.18.34+rpt-rpi-v8 #1 Debian 1:6.18.34-1+rpt1
[ 8580.170332] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 8580.170336] task:kworker/2:0     state:D stack:0     pid:62785 tgid:62785 ppid:2      task_flags:0x4208060 flags:0x00000010
[ 8580.170349] Workqueue: events drm_sched_job_timedout [gpu_sched]
[ 8580.170376] Call trace:
[ 8580.170380]  __switch_to+0xf0/0x1c8 (T)
[ 8580.170392]  __schedule+0x3c8/0xc18
[ 8580.170398]  schedule+0x3c/0x120
[ 8580.170404]  schedule_timeout+0xdc/0x130
[ 8580.170412]  dma_fence_default_wait+0x1f4/0x238
[ 8580.170420]  dma_fence_wait_timeout+0xc4/0x220
[ 8580.170427]  drm_sched_stop+0x13c/0x220 [gpu_sched]
[ 8580.170441]  v3d_gpu_reset_for_timeout.isra.0+0x68/0x120 [v3d]
[ 8580.170465]  v3d_bin_job_timedout+0x80/0xa0 [v3d]
[ 8580.170483]  drm_sched_job_timedout+0x80/0x180 [gpu_sched]
[ 8580.170496]  process_one_work+0x158/0x3b8
[ 8580.170506]  worker_thread+0x19c/0x328
[ 8580.170514]  kthread+0x14c/0x210
[ 8580.170521]  ret_from_fork+0x10/0x20
[ 8701.003627] INFO: task kworker/2:0:62785 blocked for more than 1208 seconds.
[ 8701.003650]       Tainted: G         C          6.18.34+rpt-rpi-v8 #1 Debian 1:6.18.34-1+rpt1
[ 8701.003655] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 8701.003659] task:kworker/2:0     state:D stack:0     pid:62785 tgid:62785 ppid:2      task_flags:0x4208060 flags:0x00000010
[ 8701.003671] Workqueue: events drm_sched_job_timedout [gpu_sched]
[ 8701.003699] Call trace:
[ 8701.003703]  __switch_to+0xf0/0x1c8 (T)
[ 8701.003716]  __schedule+0x3c8/0xc18
[ 8701.003721]  schedule+0x3c/0x120
[ 8701.003726]  schedule_timeout+0xdc/0x130
[ 8701.003735]  dma_fence_default_wait+0x1f4/0x238
[ 8701.003742]  dma_fence_wait_timeout+0xc4/0x220
[ 8701.003748]  drm_sched_stop+0x13c/0x220 [gpu_sched]
[ 8701.003762]  v3d_gpu_reset_for_timeout.isra.0+0x68/0x120 [v3d]
[ 8701.003785]  v3d_bin_job_timedout+0x80/0xa0 [v3d]
[ 8701.003803]  drm_sched_job_timedout+0x80/0x180 [gpu_sched]
[ 8701.003816]  process_one_work+0x158/0x3b8
[ 8701.003826]  worker_thread+0x19c/0x328
[ 8701.003833]  kthread+0x14c/0x210
[ 8701.003840]  ret_from_fork+0x10/0x20
[ 8701.003849] Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings

[mairacanal]

@Mastiffen, I'll try to reproduce the GPU hang in my setup, thanks for the detailed description! Last week, I submitted a fix for L2T cache cleaning that you might want to try: https://lore.kernel.org/dri-devel/20260707200738.659002-2-mcanal@igalia.com/

[popcornmix]

@Mastiffen if you wait about 30 mins, you can run:

sudo rpi-update pulls/7491

to get the kernel with @mairacanal 's commit in. Might be worth a test.
Read the rpi-update caveats first.

[Mastiffen]

@mairacanal I'm glad you got something out of it!

@popcornmix Thank you, I will try a bit later. No real caveats, I have all my production Pi's with full dd image backups and aslso the data on the external SSDs (docker/docker system), so the worst that can happen is that I have to go physically down in the cellar and change the SD card.

Edit: Sorry, forgot to say that I'm on my fifth dd imaging now after disabling suspend, and no crash so far. Looks promising. Is this something that survives reboots?

[popcornmix]

It doesn't survive reboots. A udev rule would be the persistent solution:

echo 'ACTION=="add", SUBSYSTEM=="platform", KERNEL=="fec00000.v3d", ATTR{power/control}="on"' | sudo tee /etc/udev/rules.d/99-v3d-power.rules

should create a rule. Then it should apply after every reboot. (check with cat /sys/devices/platform/v3dbus/fec00000.v3d/power/control).

But consider that a workaround - we'd still like to fix the root cause.

[Mastiffen]

Of course you would. And I will keep a Pi in this state for that. At the moment I just started the dd image backup of the Pi I pulled the 7491 update from.

[Mastiffen]

Sorry, @mairacanal This update did not fix it. Only disabling suspend of the driver worked.

[nbuchwitz]

Probably the same issue as #7443 (mine's on a PREEMPT_RT kernel, but same trace and same power/control = on workaround).

[mairacanal]

I was finishing to debug other issue, but I'll try to reproduce this one today. However, I already have one question for @popcornmix: I noticed that both issues are reproduced with a Raspberry Pi CM4 (not a regular RPi 4, which is the one I'm using). Is there any differences in the clock design or pmdomain for the CM4?

UPDATE: Although the user mentioned "Raspberry Pi CM4" in the issue, I see from dmesg that they are using a RPi 4 Rev 1.2 (thanks for the complete dmesg). The interesting part is that I'm using a RPi 4 Rev 1.5 and I can't reproduce the issue. From PCN, I found out that Rev 1.5 has a different PMIC, which can be relevant here.

[Mastiffen]

@mairacanal Just to make sure: You're not talking about any of my Pi's, but about other threads, right? Because I have only regular Pi's, no CM's. And mine are version 1.1 and 1.2.

[Mastiffen]

It doesn't survive reboots. A udev rule would be the persistent solution:

Sorry, @popcornmix I forgot to say thank you for the udev tip! It works, so my production Pi's are now Trixie, with that rule, working as they should.