TL;DR – rclone truly is the Swiss Army knife of cloud storage and you should look into it if you have to deal with that world at all (S3, DropBox, OneDrive, etc).

In the past I really had an addiction to Humble Bundle, where I bought a lot of cheap ebooks and comics. I dutifully put both of these into Calibre and then put it from there into my devices. Now for epubs to my e-reader, it was pretty easy. But it was quite a bit hard for comics. Eventually I forgot about all the comics.. until I bought some more. And then instead of just putting them in Calibre I put them into a folder in my pCloud storage. Then I find Panels, a really good Comic app for the iPad. But downloading a file from pCloud and then importing it into Panels was painful. I saw that the commercial version of Panels offered integration with OneDrive, GDrive, and DropBox – but not pCloud. Well I have a OneDrive that I haven't used… maybe we can move my files there? Oh but I struggle with pCloud's macOS client when I copy or rsync a lot of files out of there. It simply crashes. So I can't do all the work locally.

So that is the setup… I've had rclone on my watchlist before and mucked with it a few times but nothing serious. Well this seemed like a good way to learn about it. First I had to setup both pCloud and OneDrive – that was pretty easy. I naively decided to run a cloud-to-cloud sync, thinking that it would bomb. I was pleasently surprised! The command was:

rclone sync "pcloud:comics" "onedrive:comics"

I did this about 6pm and it finished about 11pm. I didn't think that was too bad. Oh and Panels was able to read my OneDrive and download the comic file just fine. Fantastic!

The next day I was wondering about the gigs (yes I said gigs) of other comics I had in pCloud (where I back up my Calibre) library. So I decided to copy those out of pCloud and into OneDrive

My comics are all in cbz format so I essentially had to find all my .cbz files. With rclone, this was pretty easy:

rclone ls --include "*.cbz" pcloud: > file.lst

I could have immediately copied the files to OneDrive instead of making a file list but I wanted to change the directory structure from what Calibre used. For example, I didn't want comics/My Books/Unknown/starwars legacy vol1 1412979773 (18)/starwars legacy vol1 1412979773 - Unknown.cbz but (an attempt) to do comics/starwars/starwars legacy vol1 1412979773 - Unknown.cbz. Also I really did not want to re-copy the files already existing in comics. So I wrote a quick and dirty Python script:

#!/usr/bin/env python3

from pathlib import Path
from string import Template

rclone_cmd = Template('rclone copy "pcloud:$infile" "onedrive:comics/$dirname/"')

f_list = Path("file.lst")

with f_list.open() as fp:
    for line in fp:
        size, name = line.strip().split(" ", 1)

        if int(size) < 0 or "comics/" in name:
            continue

        new_name = name
        if name.startswith("My Books/"):
            new_name = name[(len("My Books/") - 1) :]
        if " " in new_name:p
            prefix, junk = new_name.split(" ", 1)
            dirname = prefix.split("/")[-1]

        outname = name.split("/")[-1]
        print(rclone_cmd.substitute(infile=name, dirname=dirname, outfile=outname))

The result didn't really give me a directory structure that I wanted, but close. I figured I could change it manually but what I had was a good start. I also could have ran the rclone copy command in the python script, but I wanted to see the finished commands them all. So then I captured the output of my script, and then I basically had shell script. I kicked it off before bed and it finished long before I got up. And everything was copied!

So yeah… everything I wanted to do I was able to with rclone. I for sure will keep it in my toolbox.