Pro¹ X - backup, restore and repair device specific partitions

[mosen 201]

It is highly advised to take a backup of your device specific partitions right after you receive the device.
Update 20220829:
tl:dr: Community has not found a safe way to restore backups as of yet. Making below described backups is still highly advised for a future where some wizard found a reliable way to restore those backups.
For now, if you have broken sensors from flashing another OS, the restorecon method described in the "Repair sensors/ folder..." section does 100% work to fix those while retaining your attestation keys. While restoring any backup has a high chance to make the keys inaccessible in Android and can make your situation worse currently.

Ideally, backup before flashing an alternate operating system to your Pro¹ X. This has nothing to do with a data backup (e.g. using adb backup or backup apps) you might not mind too loose since you got no daily driver data on a device yet. We are about to backup all the partitions that can be needed to restore a fully working stock android. E.g. for the case you want to resell the device. Or just re decide and want to use stock android again after having tested an alternate OS.

Problemscape:
- When flashing and using SailfishOS or Ubuntu Touch (no info for LineageOS yet), the sensors folder in the persist partition gets altered. Since both OS are not in SELinux context when writting, causing the security context to break.
- This will lead to non working sensors in stock Android like described in this thread.
- The persist partition additionally contains device specific hardware attestation keys and EFS values generated during first setup of stock android.
- To be able to use Key Attestation and working sensors after a flash back to stock android, we can NOT just flash the stock persist.img since it misses our keys.
- We need a backup of persist with our keys and the correct sensor data. Ideally taken when you receive the device.
WARNING: Do not flash the stock persist manually like advised in deprecated XDA posts. You will loose your keys and a method to re-provision them is not yet found.

If you have no backup but your persist partition is broken already from booting UT or SFOS.
Check with the android app "Key Attestation" if your keys are still visible to android. It will show "Not supported by this device" to indicate your keys are not present and are sadly not restore able with the offered method. If however the app indicates a trusted environment (TEE), just follow from here on, do a backup now, saving your keys.
Then head to the last section "Repair sensors/ folder in persist.img".

This guide assumes you are using linux and have edl.py installed.
I will not be able to test this on windows. Someone else might translate the tasks to windows applications like QFIL for the flashing. The official QFIL full reflash guide does not contain this backup method yet.
@EskeRahnshows how to Backup using ADB in a comment below. Note tht this only works after rooting the device following this guide.
Details of the EDL installation and the 2.1.2 EDL image links are to be found in this "How to flash to stock" post.
You can anytime use the EDL full flash method to switch back to stock android from any soft-bricked state.
It will retain your keys since it does not flash the persist partition when it contains keys.
A full flash will however not solve the android sensor problems.

Backup of partitions
Be sure to not work in the folder you extracted the stock images too. To avoid mixing them up with your backups.
We are interested in all partitions that have potential to need a restore later, so lets grab all of these:
fsg, fsc, modemst1, modemst2 and persist.

The EDL commands to dump the partitions into image files are executed as root/sudo as following

edl r persist persist_backup.img
edl r modemst2 modemst2_backup.img
edl r modemst1 modemst1_backup.img
edl r fsc fsc_backup.img
edl r fsg fsg_backup.img

 

Restore a partition
WARNING: This method does not reliably work to restore. Only for educational purpose for now.
To fix the stock android sensor problems, we only need to flash the persist partition back.
If your persist has broken sensor config, obviously skip this step for now and do it after repairing the sensors folder.

edl w persist persist_backup.img

Repair sensors/ folder in persist.img
Root the Pro¹-X following this guide.

adb shell into the device and become root via su.

adb shell
su

Restore the SELinux security context for the sensors folder using restorecon.
First using the -n option if you just want to see what restorecon would do, but not alter anything yet.

restorecon -nvRF /mnt/vendor/persist/sensors/

And apply those changes by omitting the -n option, -v verbosity should generate the same output like when using the -n option.

restorecon -vRF /mnt/vendor/persist/sensors/

You are all set for

reboot

You should notice the fixed sensor after reboot by just folding out the keyboard and see it force landscape.

Edited September 13, 2022 by mosen

[ducksoup 110]

28 minutes ago, mosen said:

When flashing and using SailfishOS or Ubuntu Touch (no info for LineageOS yet), the sensor config in the persist partition get altered.

Is the sensor config altered merely by flashing these alternative firmwares, or is it altered only if one flashes back to Android via the full stock firmware archive?

[mosen 201]

Afaiu, its from booting into a flashed alternate OS.
While installing those, you just flash boot and userdata.
If you for some reason decide to abort right after flashing those and flash back only those two from the stock image set.
Its like nothing happened.

[ducksoup 110]

Guess that means I already toasted mine, too, since I didn't do my persist backup until after running Sailfish for a while.  😄 

Thanks for documenting the process.  I'll make sure to get that persist_backup saved right away.

[mosen 201]

I can not guarantee that above is already final since i feel a bit like a lonely tester on this.
It worked for me. But no one is around me to save me from jumping to wrong conclusions and build on them 😅

Edited August 12, 2022 by mosen

[EskeRahn 5,399]

Thank you for this thread (i added a link here)

[Hook 2,907]

2 hours ago, mosen said:

If you have no backup but your persist is altered already.
Check with the android app "Key Attestation" if your keys are still visible to android. It will show "Not supported by this device" to indicate your keys are not present and are sadly not restore able with the offered method. If however the app indicates a trusted environment (TEE), just follow from here on and do a backup now, saving your keys.

For what it's worth, Key Attestation" on my Pro1 with Lineage and Magisk shows "Not supported by this device." Not a big surprise. It has had Lineage or AICP on it since early 2020 and I've never gone back to Stock, so of course I've never noticed.  It will likely be the same with my Pro1x.  I will probably keep it on stock until Lineage is available.  I plan on using my Pro1 as my experimental device (read that learning device).  If I find a setup I really like and is stable on the Pro1, then I will try to bring it to the Pro1x.

I will see if I can backup the persist partition when I get the Pro1x.  Thanks.  I remember, even going back to Gingerbread, there was a partition you had to make sure you backed up (it had a different name, something starting with an E (not edl).  I remember I could back it up with TWRP.  Dear God, I wish we could get TWRP for this phone.  😉

[sspiff 29]

Thanks for all your research @mosen!

Unfortunately, I tried this and now my device is no longer booting into Android at all now. I haven't been able to get into recovery or fastboot mode since, only EDL mode.

When I try running EDL, it says the device is in memory dump mode and starts dumping memory from the phone. I am able to get back into the EDL flashing mode some of the time.

Behavior when booting is first showing the Powered by Android logo, then black screen. Sometimes the logo is there for quite a while.

I have an unaltered copy of my persist partition on my work PC, but for now it's late and I'm calling it for tonight.

Edited August 12, 2022 by sspiff

[EskeRahn 5,399]

I found a way to backup partitions that also works for windows, with just the usual ADB on the PC. HOWEVER this ONLY works for an unlocked device.
ADD: The below only works on Linieage/AICP not stock android, see also this

1. If not unlocked - unlock wiping data, or look for another method....
2. Make sure that your phone is in developer mode [*], and allows adb to connect as root (the phone does NOT need to be rooted)
3. Connect cable and run (first time you must accept the connection from the pc)
4. adb root
5. adb shell find /dev/block/platform -name "by-name" -exec ls -al {} \;
6. Look through the list for the partition(s) of interest. e.g.
   persist -> /dev/block/sda2
7. adb pull  /dev/block/sda2   persist_some_local_file_name.img
8. adb unroot

So reasonably simple.

[*] on 2) Settings►About, tap Buildnumber about 10 times. Back, System►Developer Options, enable "USB Debugging" and "Rooted debugging"
You can revoke one or both after step 8)..

The idea is from "way #2" here, slightly adjusted, combined with this.

 

This might not work well for all partitions on a running android, especially pushing it back, so I made an extended version also here

[mosen 201]

I removed the part with removing the sensor folder. Since when testing again now i got the same result like you had.
I must be mixing up images or something stupid... But the good news, flashing back the unaltered backup worked to solve that.

I will add a disclaimer to the Repair part above. Its obviously just for educational purpose to narrow in the solution in the current state.

Maybe we just found the reason why it takes F(x)tec so long to get the official documentation out 🤪

[EskeRahn 5,399]

10 hours ago, EskeRahn said:

This might not work well for all partitions on a running android, especially pushing it back, so I made an extended version also here

(added this, on doing it through the Recovery rather than a running android, that might not go well with all partitions, especially restore)

[mosen 201]

Latest update:
Whatever i remove or add to the partition when mounted like described above, i get a non booting device.
Even copying the known working backup sensor data into a copy of the working persist backup, leads to non boot
Flashing back the unaltered backup always works.
I highly suspect the mount method above leads to file permission problems when copying stuff.
From my PoV, all permissions look exactly the same. But the problem must lay in permissions since i copy know working things together and get a complete non boot.

I also now know a bit better how i jumped to conclusions it might be working.
I only copied the stock sensor folder onto a working backup. That continued to boot and work.
But turns out i did not fix anything since the sensors where working all along in that backup.
All i did was not destroy things further... But the conclusion is, overwriting same files seems fine and did not lead into non-boot.
Also, i can flash mounted working backup images just fine, indicating the mount alone is not the problem.

[EskeRahn 5,399]

2 hours ago, mosen said:

I highly suspect the mount method above leads to file permission problems when copying stuff.

Sounds likely when modifying inside the volume. Copying the whole image seems to work.
Just tried with 108GB userdata, via recovery and adb, and it worked just fine. Updated this.

[EskeRahn 5,399]

Frankly I think we have a better chance in patching in the keys binary in a working image, than trying to mount and fiddle inside a not functioning image with the right keys.

[sspiff 29]

I tried a couple of things:

• Copying each individual directory from the backup into the stock persist partition using `cp -a`
• Copying all of the directories apart from the `sensors` directory into the stock persist partition

All of these gave me a working phone, with rotate and keyboard backlight, but no device attestation.

`cp -a` is meant for archival backups and copies all standard properties and permissions to the destination

I'm beginning to think these device keys are either embedded in ext4 metadata not supported by standard tools, or hidden in unused data on the partition, or maybe in blocks marked as bad blocks so they can`t accidentally be overwritten.

[brunoais 317]

35 minutes ago, sspiff said:

I tried a couple of things:

• Copying each individual directory from the backup into the stock persist partition using `cp -a`
• Copying all of the directories apart from the `sensors` directory into the stock persist partition

All of these gave me a working phone, with rotate and keyboard backlight, but no device attestation.

`cp -a` is meant for archival backups and copies all standard properties and permissions to the destination

I'm beginning to think these device keys are either embedded in ext4 metadata not supported by standard tools, or hidden in unused data on the partition, or maybe in blocks marked as bad blocks so they can`t accidentally be overwritten.

Try using

ddrescue

to copy them. See if it works that way.

[mosen 201]

I got a hint from TheKit just now that it might be related to SELinux context.
He advised me to try

restorecon

on the sensor folder in a rooted adb environment.
https://www.thegeekstuff.com/2017/05/restorecon-examples/
I have no means to try for the next hours so maybe someone else is quicker.

[sspiff 29]

Prior to getting your ddrescue and restorecon tips,  I tried taking the backed up image,  delete all files, and then copy over the data from the stock persist image (hoping it would retain any hidden data but copy the stock sensor data).

This produced a non booting system.

I'll try out restorecon and ddrescue when I get back to my PC.

[sspiff 29]

Just tried reflashing my original backup and it's also failing attestation now. Not sure if I messed up somewhere and accidentally modified the file somehow (though I only mounted it read only...)

Edit: Just reflashed an older backup and it still fails attestation. Not sure what broke attestation but it seems to be outside of the persist partition? The older backup definitely predates me breaking attestation and is certainly not accidentally modified (since I keep it on removable storage and only work with copies of that file).

I will reflash the stock ROM + my persist backup to check if attestation remains broken.

Question: is it possible for my tests to have caused my keys to be revoked by whatever authority hands these out?

Edited August 14, 2022 by sspiff

[mosen 201]

30 minutes ago, sspiff said:

Question: is it possible for my tests to have caused my keys to be revoked by whatever authority hands these out?

No, as long as they remain in the data/ folder within the persist partition they are supposed to work.
I saw in other devices that the data folder containes human readable codes. But in our case its encypted gibbrish names and folders.
But generally, as long as the data folder with these keys exist. They should be recoverable.

The SELinux hint makes much sense after having read the restorecon manpages. Looks like what ever we change in the partition needs to be updated in the SELinux security layer. Afaiu now. But its Pool time now. No idea if i return to a computer today 😄

[mosen 201]

And another thing i learned, F(x)tec should be able to reprovision the devices with new keys. It seems to be a standard support issue. Afaiu, they can assign new keys as trusted source in the chain.

[sspiff 29]

17 minutes ago, mosen said:

No, as long as they remain in the data/ folder within the persist partition they are supposed to work.

Welp. I just went through the whole reflash stock procedure and restored my oldest persist backup, and attestation no longer works. It worked yesterday, which was AFTER I backed up that persist partition. Guess I'm sh*t out of luck now...

I'm not sure when the unmodified backups stopped working, I'm 90% sure I got back working attestation after flashing the backup over the stock image at least once. But I think it does invalidate my findings above, perhaps one of those operations would work. Or perhaps one of them nuked my keys. Hard to know at this point , but I guess I can't do any testing anymore now that I have no way of getting working attestation back. Sorry guys.

Edited August 14, 2022 by sspiff

[mosen 201]

On 8/13/2022 at 12:34 AM, EskeRahn said:

Make sure that your phone is in developer mode [*], and allows adb to connect as root (the phone does NOT need to be rooted)

Could you make a screenshot of where the "Rooted ADB" switch is in the Android 11 settings.
I looked through all developer settings multiply times but can not find it.
As expected, without that permission "adb root" fails with "adbd cannot run as root in production builds".

[EskeRahn 5,399]

3 minutes ago, mosen said:

Could you make a screenshot of where the "Rooted ADB" switch is in the Android 11 settings.
I looked through all developer settings multiply times but can not find it.
As expected, without that permission "adb root" fails with "adbd cannot run as root in production builds".

Ah sorry! that option is for Lineage and AICP not stock android DAMNED!!, so the whole idea crashes, and I believe there is no recovery image to boot into by stock either...
So it will require a weird approach of flashing the bootloader for LineageOS or AICP first, to fetch the Persist that way.... Then it is getting (almost) as complex as the edl

[mosen 201]

Tadi and Bru AI on Telegram are proficient in rooting the Pro¹-X already. I'll try to compile some step-by-step instructions when they showed me how to.