One of our members was facing issues with a laptop, so we teamed up.
Problem
After suspending, the laptop no longer came up and the screen remained black. Nothing helped; unplugging the main battery and the CMOS battery, unplugging AC and trying to power cycle multiple times did not do the trick.
Analysis
So we dumped the flash chip holding the current firmware and took a rough look. The laptop is based on an AMD platform, and it could be parsed fine.
We then searched for a firmware upgrade from the vendor. Our first attempt led us to the wrong product, because there is also an IdeaPad 330-15ARR (mind the missing S).
The correct product website is this one: https://pcsupport.lenovo.com/us/en/products/laptops-and-netbooks/300-series/330s-15arr
Via “Drivers & Software” -> “Manual Update” (Select Drivers) -> “BIOS/UEFI (1)”,
we ended up on the firmware download page, and picked “BIOS Update” with a size
of 7.12 MB and the name 7WCN38WW, with the date 08 Jan 2020.
That gave us 7wcn38ww.exe.
Unpacking
From earlier experience, we knew that innoextract 7wcn38ww.exe would extract
it for us, and we got a directory named app/ containing the file BIOS.cap.
That file is larger than the flash chip, because it is an upgrade image, a
UEFI application containing the actual image.
Recovery
In various forums, including the official community forum, we read that putting the upgrade file on a USB stick and pressing Fn + R on boot would start a recovery process. However, all we saw was the power LED blinking then, and nothing else seemed to happen.
Extracting
So we looked at the upgrade file BIOS.cap again.
Using xxd, we could quickly find the start of the raw firmware image, and
using dd and truncuate, we cut it out and truncated it to 8 MB:
dd if=app/BIOS.cap bs=8 skip=99 of=fresh.rom
truncate -s 8M fresh.rom
Via biodiff, we compared it against the backup, seeing that it looks very
similar. So we gave it a try and flashed the fresh image to the laptop.
Almost there
We then tried to boot again, and we were very pleased to see the Lenovo logo, followed by the OS bootloader. But the OS failed to boot, showing a black screen only, and no kernel output. So we went into the firmware settings, and saw that the product name was set to “ILLEGAL”. Weird.
An attempt to boot a fresh Linux Mint image from a USB stick worked though. So we installed that on a new partition, rebooted, but once again, the kernel did not get far. With extra options, the kernel stub at least dropped us two lines, but that was it.
Another fix
So we compared the firmware image that we flashed against the backup again. And we saw that there was what looked like the product name in the backup (left), but not in the fresh image (right):
001e dff8 ff ff ff ff ff ff ff ff |........ | 001e dff8 ff ff ff ff ff ff ff ff |........
001e e000 4c 65 6e 6f 76 6f 20 69 |Lenovo i | 001e e000 ff ff ff ff ff ff ff ff |........
001e e008 64 65 61 70 61 64 20 33 |deapad 3 | 001e e008 ff ff ff ff ff ff ff ff |........
001e e010 33 30 53 2d 31 35 41 52 |30S-15AR | 001e e010 ff ff ff ff ff ff ff ff |........
001e e018 52 00 00 00 00 00 00 00 |R....... | 001e e018 ff ff ff ff ff ff ff ff |........
001e e020 00 00 00 00 00 00 00 00 |........ | 001e e020 ff ff ff ff ff ff ff ff |........
001e e028 00 00 00 00 00 00 00 00 |........ | 001e e028 ff ff ff ff ff ff ff ff |........
We patched that part into the raw flash image:
dd if=backup.rom bs=4096 skip=494 count=1 | \
dd of=fresh.rom obs=4096 seek=494 conv=notrunc
And we finally saw the product name show up in the firmware menu, the kernel happily booted, and the owner of the laptop is happy again.
Recap
Since we assumed firmware corruption, we took another look and compare the fresh firmware image against the backup again, seeing that indeed, there were many blocks of 64 bytes each that had been cleared:
002f 0350 ff ff ff ff ff ff ff ff |........ | 002f 0350 ff ff ff ff ff ff ff ff |........
002f 0358 ff ff ff ff ff ff ff ff |........ | 002f 0358 ff ff ff ff ff ff ff ff |........
002f 0360 ff ff ff ff ff ff ff ff |........ | 002f 0360 ff ff ff ff ff ff ff ff |........
002f 0368 ff ff ff ff ff ff ff ff |........ | 002f 0368 ff ff ff ff ff ff ff ff |........
002f 0370 ff ff ff ff ff ff ff ff |........ | 002f 0370 ff ff ff ff ff ff ff ff |........
002f 0378 ff ff ff ff ff ff ff ff |........ | 002f 0378 ff ff ff ff ff ff ff ff |........
002f 0380 65 00 66 00 61 00 75 00 |e.f.a.u. | 002f 0380 ff ff ff ff ff ff ff ff |........
002f 0388 6c 00 74 00 43 00 72 00 |l.t.C.r. | 002f 0388 ff ff ff ff ff ff ff ff |........
002f 0390 65 00 61 00 74 00 65 00 |e.a.t.e. | 002f 0390 ff ff ff ff ff ff ff ff |........
002f 0398 46 00 6c 00 61 00 67 00 |F.l.a.g. | 002f 0398 ff ff ff ff ff ff ff ff |........
002f 03a0 00 00 01 ff aa 55 3f 00 |.....U?. | 002f 03a0 ff ff ff ff ff ff ff ff |........
002f 03a8 07 00 00 00 00 00 00 00 |........ | 002f 03a8 ff ff ff ff ff ff ff ff |........
002f 03b0 00 00 00 00 00 00 00 00 |........ | 002f 03b0 ff ff ff ff ff ff ff ff |........
002f 03b8 00 00 00 00 00 00 00 00 |........ | 002f 03b8 ff ff ff ff ff ff ff ff |........
002f 03c0 ff ff ff ff ff ff ff ff |........ | 002f 03c0 ff ff ff ff ff ff ff ff |........
002f 03c8 ff ff ff ff ff ff ff ff |........ | 002f 03c8 ff ff ff ff ff ff ff ff |........
002f 03d0 ff ff ff ff ff ff ff ff |........ | 002f 03d0 ff ff ff ff ff ff ff ff |........
002f 03d8 ff ff ff ff ff ff ff ff |........ | 002f 03d8 ff ff ff ff ff ff ff ff |........
002f 03e0 ff ff ff ff ff ff ff ff |........ | 002f 03e0 ff ff ff ff ff ff ff ff |........
002f 03e8 ff ff ff ff ff ff ff ff |........ | 002f 03e8 ff ff ff ff ff ff ff ff |........
So we conclude that some race condition may have occurred when the firmware tried to handle the suspend request, but since it is closed source, and we did not intend to spend weeks and years reversing all the components that are part of the power management, we are leaving it at this point.
Given that the last upgrade was from 6 years ago, we assume that the vendor would not offer a fix for this anymore, and so we hope it won’t occur again, though this has happened to many other people, looking at posts in both the official community forum and also other places.