Jump to content
Sign in to follow this  
david

File transfers (wireless and wired)

Recommended Posts

The purpose of this thread is to share tips and experiences on transferring files between phones and other devices, whether through wired or wireless connections.  Whether you use USB or WiFi or Bluetooth or FTP or WebDAV or SMB or something else, please share your solutions.

This started out as a conversation in another thread.  @EskeRahn has moved the applicable posts to this thread.  The rest of this post was the first post in the old thread that started talking about this topic:

...

17 hours ago, Hook said:

That looks exactly like mine. also USB 2, but don't care for the same reason @Craig doesn't care.  Large file transfers are pretty rare for me, so even the 2,0 speeds are fine with me.

I would certainly be using it for charging more than file transfers, and I can do most file transfers over a slow USB connection or wifi, but there are times when it sure would be nice to have that extra speed, when I'm in a hurry.  But the answer to that is just to get a non-magnetic cable and unplug the magnetic tip and use the fast cable for those rare situations I guess.

I'm curious what the max real-world speed is that anyone has gotten out of the Pro1 when transferring data from the phone to a PC over the USB-C connection with a faster version of USB?

Edited by david

Share this post


Link to post
Share on other sites

FTP over wifi is probably faster and more reliable than anything you get over that shoddy MTP over USB to be honest. If you had mass storage mode, that would be a different story of course.

  • Thanks 2

Share this post


Link to post
Share on other sites
4 hours ago, elvissteinjr said:

FTP over wifi is probably faster and more reliable than anything you get over that shoddy MTP over USB to be honest. If you had mass storage mode, that would be a different story of course.

Regarding WiFi, I've had good luck using a WebDAV server on the phone and a WebDAV client on the pc, that allows me to map it as a drive letter.  I then use a special file transfer application on the PC that is multithreaded and works well for higher latency connections.  I can post the specifics if people are interested.  

I haven't tested the speed of that approach to FTP yet, but maybe I'll try that at some point.  Not all WebDAV servers are created equal.  I had some larger files at one point that other apps would fail on.  I had almost given up, but then found a simple server app that could handle them, so that is what I've used ever since.

I had to take the wifi approach, because at the time, my phone's USB port stopped working for data transfer and would only work for charging.

I'd still be interested in real world USB 3.0 (also known as USB 3.1 Gen1 (although if someone knows if it is Gen2, please let us know)) speeds to see if they are truly slower than wifi.  And, of course, with wifi, if the PC is also on wifi, then depending on how the phone and PC are connected to the router, the speeds can be lowered due to sharing the same frequencies and the half duplex nature of wifi.  I have a hodgepodge of PCs, with some only supporting 802.11N and some supporting 802.11AC.  The capabilities of the router and client, with regards to number of antennas/streams/MIMO/bandwidth also play a large role in the speed that can be achieved.

Is there not a way to enable mass storage mode with root on modern androids?

Edited by david
  • Like 1

Share this post


Link to post
Share on other sites

Running a ftp server on phone is easy cuz all PCs support it, you can add it as a favorite/bookmark to file manager in windows or linux or anything else, without installing anything on the PC end.

I use this one, super easy to configure, even has an option to auto-connect when you're on home wifi, so basically whenever you're at home your phone is available in your PC file manager.

https://f-droid.org/en/packages/be.ppareit.swiftp_free/

(fyi, fdroid version is same as full version from playstore, even tho it says free.)

  • Thanks 3

Share this post


Link to post
Share on other sites
Just now, Craig said:

Running a ftp server on phone is easy cuz all PCs support it, you can add it as a favorite/bookmark to file manager in windows or linux or anything else, without installing anything on the PC end.

WebDAV is built into Windows too.  You can map it to a drive letter the same way you can an FTP server, in explorer, without installing anything.  I don't remember if it was the performance or if it was flaky in some way, but I elected to pay for a product to take the place of the built in WebDAV client in Windows.  But it is there, and probably works for most use cases.

  • Like 1

Share this post


Link to post
Share on other sites

As mentioned previously, I use a WebDAV server on my phone and a WebDAV client on my PC to map the phone as a drive letter.  I then use an optimized file copy application on the PC to do the copying, for maximum speed over a wifi connection.  There are a lot of variables here (phone hardware speed, phone software speed, wifi network speed (a lot of variables just in that area), PC speed, etc.).  I use the above for both speed reasons and to transfer large files. 

I've only found one WebDAV server for android that doesn't choke on very large files (in my case the files were approximately 4GB and 3GB).  The other servers may have also had problems with really deep directory structures.  And the built in WebDAV client in Windows (accessible by mapping a drive letter from Windows explorer, for instance, which is called WebDAV Mini Redirector) also chokes on large files and is slower than other options.  It can be used in a pinch, for small transfers, I believe, but I'd recommend using another client.

NOTE:  All the above is with Windows 7 on the PC and Android KitKat on the phone (technically Cyanogenmod equivalent).  And the phone is a Samsung Galaxy S Relay 4G.

I should also mention that I used to push files to the PC via apps on the phone that can mount SMB shares on the PC.  I'd mount the admin shares (C$, D$, etc.) with the software and sometimes start pushing files when I went to bed.  This approach does work, but was about 5 or 6 times slower than the solution I ended up with.

 

Here are the pieces I use for the WebDAV solution:

This used to be in the Play Store, but appears to have been removed.  The developer's website still has documentation available for how to use it.  There may even be some tricks in there that I never tried that could make the built in Windows WebDAV client work better.

 

This is a pay product, but does have a free trial.  It supports Windows and Mac and supports mapping drive letters to cloud products, like Google Drive, Amazon, DropBox, etc. 

 

Another option, that has a free version, that I haven't tried yet, is RaiDrive.  It seems to have similar features to WebDrive, but does have the free version that WebDrive doesn't.  The pay version has some sort of performance improvement when using Windows File Explorer to copy files.

 

This is a key part of maximizing speed.  Using Windows Explorer to copy files is sloooooow.  This tool is free and is optimized for speed.  There are a ton of settings and filters that can be applied for doing copies.  I believe some people have had issues with it not running well on certain server versions of Windows and there could possibly be some bugs with paths longer than 255 characters.  The comments on the download page also give some pointers to other copy applications that can be tested to get around these issues.  I haven't tested them, but if they work as fast, then I might consider replacing RichCopy with one of those for my uses.

 

NOTE:  Due to a limitation in DaveDrive Lite, only the internal storage can be mapped, not an external SD card.  To get around this, you have to create a bind mount from the internal storage to the external storage.  I use FolderMount (requires root) to do this.  I have a folder under /SDCARD called SDCARD1 and that points to the external SD card.

Apparently this hasn't been updated to work with Magisk, according to a lot of the comments.  It may have problems with newer versions of Android too, but it is unclear if that is for certain vendors or all vendor implementations.  When I get my Pro1, if I can't find a WebDAV server that can handle large files, like DavDrive Lite, I'll probably have to script the bind mounting to happen automatically from the internal storage to the external SD card.

 

Other notes:

  • I moved the WebDrive cache folder from C:\ (somewhere down in my users folder) to another drive.  Windows will still use c:\windows\temp for its own caching, but at least it won't use up twice the space by also having the file go through the cache for WebDrive on the C:\ drive.  I did this, because my C:\ drive is an SSD drive with limited space.
  • Watch for low space on C:\.  Try to keep at least several GB free before doing a transfer if you have a lot of data you're transferring from you phone.
  • Use Beyond Compare on Windows to compare folders/files.  Use the session settings to set it to compare the contents using CRC (bit by bit binary is too slow and not needed).  There is a free trial version.  This is a very nice compare tool, regardless of what method you use to get files off your phone.  This will allow you to be sure that nothing that was copied had any corruption and it will also tell you which files/folders might have been missed from the source folders on the phone.  The UI takes a little getting used to, but it is very powerful.
Edited by david
  • Like 1

Share this post


Link to post
Share on other sites
On 1/14/2020 at 8:29 PM, silversolver said:

What are we defining as large file transfers, exactly? At 480mbps, copying 1GB takes 20-30 seconds depending on your USB controller--no, they're not created equal! I can move tons of junk in a few minutes.

Makes me think of this: https://www.gocomics.com/calvinandhobbes/1995/08/17

Screenshot_20200114-213128.png

Modems and car phones. :-)

The 480 Mbps is the theoretical top speed.  As you mentioned, the controller and other factors may impact this.  MTP is a slow protocol.  It doesn't allow for parallelism.  The hope is that USB 3.0 will make up for some of that.  If you can't do multiple operations concurrently, then try to do each serial operation as fast as possible (that's the idea). 

Of course, there is also the speed of the internal storage or SD card storage or PC storage at play here too.  Lots of variables.

I've never used a USB 3.0 phone to do data transfers, so I don't know how it will work in practice.  My Galaxy S Relay 4G only has USB 2.0.  Another family member's Galaxy S6 only has USB 2.0 (for some reason Samsung went from USB 3.0 on the S5 to USB 2.0 on the S6!).  A couple family members do have Pixel 1 phones, so I guess I can test with one of those (they support USB 3.0).  They just upgraded to Pixel 4XLs, so I can also test with one of those sometime, as well.  Those are USB 3.1, but the specs I can find online don't state if it is Gen 1 (same as USB 3.0) or Gen 2 (10 Gbps).  The internal storage and processor will be faster, which should help things as well.

But to really answer your question, for me, I am talking about doing backups of a lot of photos/videos or doing backups of a the phone's storage (of which, the bulk of the space is being taken up by photos/videos, but there can be a lot of smaller files as well).  In most cases, I'm not in a hurry, so it doesn't matter.  But then there are times when I realize I need to make space right before going to a family event and I'd rather be able to transfer them faster than slower.  Or cases where I'm spending a lot of time trying to fix a phone and it involves copying all the files back and forth multiple times (as was the case when the S6 in the household got into a boot loop a while back).

Now, there are ways around the MTP issue, to get closer to USB 2.0 theoretical speeds.  I used ADB commands to pull/push files for the S6 in question.  But with 128 GB of storage, that still took an hour+ for each attempt to transfer the files.  And ADB pull has bugs that cause it to hang periodically, so it requires running it many times to get it to complete, and that means everything takes longer, so every bit of speed I can get out of the port is time saved that I can spend with my head on a pillow. :-)

While googling around, I did find a new technique that I hadn't thought of before.  It involves using USB tethering to connect the phone to the PC and then transferring files over that connection with different apps (including the WebDAV or FTP route that we've talked about in other posts).  I'll have to try that sometime.

https://android.stackexchange.com/questions/91900/is-there-a-viable-alternative-to-mtp-for-file-transfer

With an external SD card, it can always be taken out and put into a USB adapter and copied that way.  I zapped one once by doing that, so I vowed not to do that with my Relay 4G again unless entirely necessary.   (I don't think I lost any data.  I think it was after I got the data off. The thing just lost its file format and I had to reformat it or repartition it or something.) 

I did this recently with my mom's Galaxy S7, because their computer wouldn't recognize the phone over the USB connection and the wireless options I tried at the time weren't working and she had a ton of photos/videos on that SD card.  Again, another case where I was under the gun and it was taking a long time to complete the task, when I didn't have that time to spend. 

With the Pro1, from what I've heard, the SD/SIM tray isn't robust, so I don't think I'm going to be opening it a lot to do data transfers.

But yes, the normal plugging it in via USB and copying a few files with Windows Explorer works fine (if Windows recognizes the phone, which is another fun challenge).  It is when we get into thousands of files and many GBs, and time crunches, that the extra speed would be nice.

Edited by david
  • Like 1

Share this post


Link to post
Share on other sites

Most USB 3.0 thumb drives are faster than USB 2.0 thumb drives.

In fact, they're fast even when plugged into USB 2.0 slot.

There are always bottlenecks somewhere.

Even tho I only have USB 2 ports on my old laptop, I only buy US3 thumb drives.

But cmon, we're in the future, use wifi to transfer files between pc & phone.  Cables take effort.  Bluetooth/IR are slow.

 

For most people, ftp is gunna be more than enough.  No need to deal with complicated stuff.

Edited by Craig
  • Like 1

Share this post


Link to post
Share on other sites
3 minutes ago, Craig said:

But cmon, we're in the future, use wifi to transfer files between pc & phone.  Cables take effort.  Bluetooth/IR are slow.

 

For most people, ftp is gunna be more than enough.  No need to deal with complicated stuff.

I agree.  "For most people."  I think we need to differentiate between edge cases and the typical cases.  I'm looking more at the edge cases.  For the typical cases, a lot of solutions will work fine.

When time allows, I might do some speed testing to show the difference between the various options, with different numbers of files and different amounts of data.  That will be easier for people to relate to and pick from based on their use case.

In the meantime, at least we're giving people options.

  • Like 1

Share this post


Link to post
Share on other sites

A bit off topic, but talking MTP-filetransfer you might be interested in this for doing incremental transfers. I use it for rather efficiently fetching copies of new screendumps, photos and other.

Note that not all can be seen through MTP (e.g. SMS and MMS missing), so use it combined with a backup tools EXcluding photos and the like, as these tools typically can only do total backups.

Share this post


Link to post
Share on other sites
1 hour ago, EskeRahn said:

A bit off topic, but talking MTP-filetransfer you might be interested in this for doing incremental transfers. I use it for rather efficiently fetching copies of new screendumps, photos and other.

Note that not all can be seen through MTP (e.g. SMS and MMS missing), so use it combined with a backup tools EXcluding photos and the like, as these tools typically can only do total backups.

Good find.  I think that is right on topic.

I am hoping to install an app that will take care of incremental transfers for me by automatically (over wifi) keeping my photos and videos synced to a given PC.  The app is called Syncthing and has an Android component and then components for the PC end (supports Mac, various forms of Linux, Windows, etc.).  I'll have to jump through some hoops with some settings so that if I delete files form the phone, it doesn't delete them from the PC and if I delete them from the PC, it doesn't delete them from the phone, but I'm hoping this solution will cut down on having to do all these massive file transfers, except in the case where I need to back up a phone to factory reset it and restore it.  I'd still like to take periodic nandroid backups (once those are possible on the Pro1, with TWRP, possibly), but I don't need to take those as often. 

Photos and Videos are number one on the list of things not to lose for me.  Then there is app data.  Then there is the system and the apps installed, so that I don't necessarily need to start from scratch if something goes wrong.  I might even be able to get Syncthing to copy some of the app data to the PC with root and with symbolic links or something.  Android has gotten so picky about only apps being able to access their own data, etc., so things like this are more difficult now.

Edited by david

Share this post


Link to post
Share on other sites
On 1/14/2020 at 8:52 AM, elvissteinjr said:

FTP over wifi is probably faster and more reliable than anything you get over that shoddy MTP over USB to be honest. If you had mass storage mode, that would be a different story of course.

Correct. I typically transfer large files between my BlackBerry KeyOne and PC with SFTP. If I'm not too far from the router I get speeds approaching 50 MB/s with large files in either direction, and at that point I believe it's limited by the read/write speed of the eMMC chip in the phone. Of course a directory full of smaller files runs slower, but even then it's much faster than with the USB MTP connection.

On 1/15/2020 at 1:22 AM, david said:

Photos and Videos are number one on the list of things not to lose for me

Google Photos backs that up to your account in the background and it all happens seamlessly. For me it's peace of mind that irreplaceable memories won't be lost if my phone is damaged or lost/stolen as they'll always be stored in my account.

Share this post


Link to post
Share on other sites
3 minutes ago, jcz95 said:

Correct. I typically transfer large files between my BlackBerry KeyOne and PC with SFTP. If I'm not too far from the router I get speeds approaching 50 MB/s with large files in either direction, and at that point I believe it's limited by the read/write speed of the eMMC chip in the phone. Of course a directory full of smaller files runs slower, but even then it's much faster than with the USB MTP connection.

That's some good speed (400 Mbps).  Is that what is reported as the link speed or is that what the SFTP client is reporting as the transfer speed? 

Can you share your wifi parameters?  802.11AC?  PC on wifi too?  Channel width?  Number of streams?

Which SFTP app are you using on the phone and which application on the PC?

Any stats on the USB MTP connection speed?  And was that with a USB 3.0 port and cable? (I read some reports online of the KeyOne shipping with a USB 2.0 cable instead of a USB 3.0 cable).

3 minutes ago, jcz95 said:

Google Photos backs that up to your account in the background and it all happens seamlessly. For me it's peace of mind that irreplaceable memories won't be lost if my phone is damaged or lost/stolen as they'll always be stored in my account.

That's great if you pay for it.  Without paying for it, once you go over your free quota, it lowers the quality of the photos and videos.  They may still be of acceptable quality, but I like to save mine in the original, full quality.

With the Pixel 1 phones in our house, they have lifetime, full-quality google photos backup.  The Pixel 4 XLs we just bought do not have that benefit. 😞

I do agree that background transfers are a very good option (whether to the cloud or to a local machine).

Share this post


Link to post
Share on other sites

If you have a linux box on your LAN (or know enough windows to get a server working) FolderSync works well.

I use it for two-way syncing. 99% of the time, it is effectively just a backup tool, copying everything from the (internal and external) sdcards to my linux server. Occasionally, when I want to put something onto the phone, I just put it in the appropriate folder on the server and run a sync. I mostly have stuff on that server in the first place, so it's simple for me.

LAN recommended... it operates per file, so the overhead can become prohibitive on a slow connection.

Share this post


Link to post
Share on other sites
1 hour ago, zurvan2 said:

If you have a linux box on your LAN (or know enough windows to get a server working) FolderSync works well.

Thanks!  It looks like running it on the phone would also allow syncing to SMB shares on Windows.

In that use case, I did google it and found a report of high battery usage, although that was on an older version of Android.

1 hour ago, zurvan2 said:

I use it for two-way syncing. 99% of the time, it is effectively just a backup tool, copying everything from the (internal and external) sdcards to my linux server. Occasionally, when I want to put something onto the phone, I just put it in the appropriate folder on the server and run a sync. I mostly have stuff on that server in the first place, so it's simple for me.

LAN recommended... it operates per file, so the overhead can become prohibitive on a slow connection.

Do you also use it to backup your photos/videos on your phone?  If so, how does it deal with trying to sync deletes (if you don't want it to do that)?

Share this post


Link to post
Share on other sites

You can choose whether or not to sync deletes per "folder pair".

I have one "folder pair" set up for all of the SD card, with an exclusion for DCIM, and a separate folder pair for DCIM with slightly different rules. You can repeat this pattern and have completely separate rules for any folder.

The initial sync is definitely expensive. Once the initial sync is complete it's much better. The whole thing is not as efficient as rsync or unison, but it does work.

  • Thanks 1

Share this post


Link to post
Share on other sites
On 1/16/2020 at 4:45 PM, david said:

That's some good speed (400 Mbps).  Is that what is reported as the link speed or is that what the SFTP client is reporting as the transfer speed? 

Can you share your wifi parameters?  802.11AC?  PC on wifi too?  Channel width?  Number of streams?

Which SFTP app are you using on the phone and which application on the PC?

Any stats on the USB MTP connection speed?  And was that with a USB 3.0 port and cable? (I read some reports online of the KeyOne shipping with a USB 2.0 cable instead of a USB 3.0 cable).

That speed is what the SFTP client reports.

The phone is connected over 802.11ac to a Synology RT2600ac and the PC is wired. Channel width is 80 MHz. Not sure where I can see the number of streams since it isn't shown in the router configuration pages or the WiFi analyzer app I'm using on my phone.

On both the phone and PC I'm using the SFTP/SCP client built into the OpenSSH package.

As for the MTP connection speed I don't have solid numbers since Windows doesn't show transfer speed in the MTP file operation status, just a progress bar and estimated time remaining (which is never accurate). That said I am using the cable that came with the phone so it's limited to USB 2.0 which is comparatively slow.

Share this post


Link to post
Share on other sites

Thanks for providing the details.

1 hour ago, jcz95 said:

The phone is connected over 802.11ac to a Synology RT2600ac and the PC is wired. Channel width is 80 MHz. Not sure where I can see the number of streams since it isn't shown in the router configuration pages or the WiFi analyzer app I'm using on my phone.

My guess is that you have a single stream.  I believe that the KeyOne is limited to a single stream due to only having a single antenna.  My guess is also that the upload speed from the KeyOne to the router would be slowed down if other devices are transmitting at the same time on that channel, since you need 2 antennas with MU-MIMO to participate in it.  That wouldn't affect downloading to the phone though, only uploading.

The theoretical max for a single stream of 802.11ac, with 80MHz of bandwidth, is 433 Mbps.  You are doing very well to get close to that theoretical limit.  In theory, you could get twice your current speed with the same set-up on the Pro1, since it should support 2x2 MU-MIMO, for 800+ Mbps.  If you are able to run on a 160MHz channel, you could get 4 times your current speed, for 1600+ Mbps.  Our Pro1s won't support 160MHz though, and I doubt the router could keep up with that speed either.

The absolute maximum you can get out of an external SD card in the Pro1 would be 800 Mbps (reading, which means sending from the phone).  Real world max would probably be more like 600-700 Mbps, with a fast SD card.  Sending to the phone, we'll probably be lucky to get 500 Mbps, due to the slower write speed of the cards. 

Theoretical max for the internal storage would be much faster.  The UFS 2.1 spec allows for very fast speeds, but it is unknown how fast the chips are that fxtec used.  My guess is that the phone can probably exceed the router, when transferring data over wifi.

To that point, I'm guessing your router (and mine too, an ASUS RT-AC5300) will be the limiting factor when transferring to/from the Pro1.  I'm going to guess, based on online info, that our routers will max out at between 400-500 Mbps.

NOTE:  All the above theoretical speeds are without overhead of file transfer protocols.  Those are just raw data speeds.

1 hour ago, jcz95 said:

As for the MTP connection speed I don't have solid numbers since Windows doesn't show transfer speed in the MTP file operation status, just a progress bar and estimated time remaining (which is never accurate). That said I am using the cable that came with the phone so it's limited to USB 2.0 which is comparatively slow.

I'm still curious how big of a difference a USB 3.0 cable will make and whether it can overcome some of the MTP inefficiency.  You're right -- the testing is harder with MTP.  We'd have to use a stopwatch and do the calculations ourselves.

The wifi method is fast, but it will still take 45+ minutes to transfer 128 GB of data at 400 Mbps.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Sign in to follow this  

×
×
  • Create New...

Important Information

Terms