Features similar to Multcloud?


#1

Hi team,

Just want to ask if there is anything on the roadmap (which I’d be interested in seeing if it’s public somewhere and I missed it?) with regards to allowing some of the following functions that I currently pay Multcloud for:

Data Migration: Give it a source and destination folder, either on the same or different cloud providers, and it’ll move the data from one to the other using APIs. Not completely robust, and has a LOT of issues with Amazon, but I’ve stopped using that altogether now since the issues they’ve been experiencing.

Multiple Sync Options: Currently MultCloud does this as a “periodic job” that pulls/pushes files that have changed. It’s lacking delta-transfers or real-time sync. I have included the descriptions they give for each type below:

  • Two-Way Sync: The added, modified and deleted files in the source directory will be replicated to the target directory. Likewise, the added, modified and deleted files in the target directory will be replicated to the source directory (Namely, the files in the source and target directories are always the same).

  • One-way Simple: Sync via the simplest way:
    The added, modified and deleted files in the source directory will be replicated to the target directory. However, the added, modified and deleted files in the target directory will not be replicated to the source directory.

  • One-way Mirror: Sync via mirror:
    The files in the source and target directories are always the same (MultCloud will delete the extra files in the target directory).

  • One-way Move: Sync via move:
    Once the synchronization is completed, all the files in the source directory will be deleted.

  • One-way Cumulative: Sync via cumulative:
    When deleting the files in the source directory, the same files in the target directory will not be deleted.

  • One-way Update: Sync via update:
    All the files in the target directory will be deleted firstly, and then the added and modified files in the source directory will be transferred to the target directory.

  • One-way Incremental Backup: Sync via incremental backup:
    Each time when you sync files, a subdirectory will be created in the target directory so that the added and modified files in the source directory will be transferred to it.

  • One-way Full Backup: Sync via full backup:
    Each time when you sync files, a subdirectory will be created in the target directory so that all the files in the source directory will be transferred to it.

While I do not use all of the these options (currently only Move and Full Backup), they all have uses I can easily envision. Further, it would probably be possible to simplify this list into a series of selectable yes/no options for synching each type of change (create/update/delete/etc) each way.

I would obviously be willing to pay more to have access to these features, since I am currently paying for this now.

Thanks


Announcing odrive2
Announcing odrive2
#2

Hmmm… interesting.

Well, we’re definitely going to have Data Migration as a simple feature (drag and drop your file between two of your linked storage folders). It will perform the migration using your desktop client as the intermediary / broker.

With regards to the Sync Options, we’ll have something similar, but our expression will be much simpler. We’ll have these as available (paid) options:

  • Sync (including the placeholder sync capability that you saw in odrive1) which lets you set up direct sync relationship between any folder on your computer with any remote storage folder you’ve linked on odrive2.
  • Backup which syncs up changed files from local to remote but won’t sync deletes and will keep prior versions of updated files.
  • Backdown which is just the same as Backup but in case you want to backup from remote to local for disaster recovery or other purposes.

That would cover most of the common use cases people may have, in our opinion. But I hope people will engage with us and describe anything else that might be important to them.

Thanks,
-Jeff


#3

The key thing I think is missing from how I read your post is that there will be no remote-remote sync option.

Specifically in my case: Having a Backup option (possibly with optional “remove source on completion” to help with logs/uploads) which takes one cloud storage location and shoves it into another without using the client to pull down and back up is definitely attractive. I could host a cheap VM to perform this function, or use a service such as Multcloud, however I feel it is something that would be best approached in ODrive - given its positioning as a “Multi-cloud management layer” application.


#4

Yes. That’s currently correct. It’s been a possibility that we’ve discussed for a while, but there are no active plans to build a remote-remote sync (in this case, I’d call continuous migration) service yet. If there’s a lot of immediate external interest in it, of course we’d be happy to make plans to build it faster. =)

As always, thanks for your input! You’re very knowledgeable about what you want and what possible solutions could achieve it.

-Jeff


#5

I recognise that I am not a “typical user” though, and that a lot of my use cases are very specific or aren’t “normal” when compared to general internet users. That said, I am equally happy to “roll my own” in the event that something is not readily available through commercial means.

My hope with beta participation is that, even if there is no scope to direct any development, at least I will have a more intimate knowledge of the product and its features and limitations such that I would better understand what I could build on top, or beside, it without breaking it.

Specifically in relation to this thread - I have previously looked at using AWS to build a cloud-to-cloud sync product, however that was shelved once I discovered Multcloud - however, their product isn’t without its quirks either.


#6

Thanks for the feedback! I am interested in supporting the use cases. We are gathering market information to properly design and package the service. Are you managing your own data, your company’s data, or your customers data? Can you share your use cases (why and what) in general?

-Peter


#7

I will gather some specifics of what I can share with regards to the setups I am involved with and post back shortly.

Edit: “Shortly” became a lot longer than anticipated, sorry!


#8

So, to clarify my use cases. I currently use cloud storage for a number of things, in all three of the examples you have provided. Please excuse some of the vague information in some of the below, as I am not permitted to share some specific implementation details.

Personal Data

For my personal data, I use a combination of GDrive, OneDrive and Dropbox. They all serve their own purposes.

  • Dropbox is for “general use” and is mainly for storing my PDF libraries, media files, portable apps, etc. I also use the “File Requests” feature of Dropbox to have other people send me things easily.
  • OneDrive is for “Documents” - it integrates natively with my Azure/365 device management for My Documents and such
  • GDrive is for “Sharing and collaboration” - it’s used mainly for GDocs, and also for photo backup (autosync from phones/etc)

There is some overlap in the uses above, it’s not completely delineated, but that’s the general use cases of those drives. Where ODrive currently fits in, is to allow access to GDrive without a client (the Google Sync client is terrible), but also to allow access to secondary Dropbox/OneDrive accounts. These can include:

  • Drives/folders shared with me as part of a contract
  • Secondary free 2GB Dropbox accounts for specific use cases, usually archival data (old chat logs, expired contracts, artwork, etc)
  • Synching folders from AppData and other similar locations into some cloud storage - Note: In most cases I will use Windows Junction Points for this, due to previous issues I have had with ODrive Sync, but it is still used in some cases - mostly where the cloud account to sync to is a secondary one (above) and not a local folder.

Multcloud is currently in very limited use - mostly just used to manually move folders (old client data, etc) to archival accounts when required, without synching down and back up, or requiring logging into multiple cloud drive clients. I also used it to migrate away from Amazon Cloud Drive recently, by simply pointing it at the root and saying “Put this into a subfolder in my Dropbox, go.”

My Company Data

I am using a combination of Dropbox and OneDrive/Sharepoint to store company data currently, and using Multcloud to ensure data redundancy by synching between clouds. Dropbox is also supported on Linux, so I am able to have cron jobs push data into a folder, Dropbox sync it up, MultCloud “additive move” it to another account, and then Dropbox removes it from the local drive. This helps with automated backups on some very small Docker instances I use primarily.

Customer Data

A big use case for this is a current client that uses Dropbox Free (2GB) accounts all with folders shared back to a central “Main” account, which is paid, and stores all data. Previously, archival data was manually copied out to unshared folders to prevent it being overwritten - this is now automated with Multcloud. Additionally, there is now a “Dev” environment created each night as a snapshot of their data, which is pushed into GDrive (via MultCloud) and accessed by some of their external staff (they use GApps primarily).

There are also smaller uses at my clients for this kind of sync, which would probably be able to be achieved in other ways, but are able to be done simply with the options Multcloud presents. As ODrive is something that could potentially help with the (somewhat lacking in some cases) UX of Cloud Storage Providers in general, especially the web portal, I’d like to think that an entirely “client-less” use case is possible.

Sorry for the long post, I hope this helps. Feel free to shoot me questions about any of it and I’ll answer them the best I can!


#9

Fantastic post. Thank you for a well thought out response!

I can see the “data orchestration” requirements for backup, archive, arbitrary workflows. A client-less solution would be very convenient.

Can I ask you a couple more questions to clarify context?

In your professional capacity, do you work in an IT department or as with an IT service provider? Alternatively, more generically, who do you think has these problems and can be customers of an odrive solution?

thanks,
-peter


#10

I personally am an IT and business improvement consultant, working as my own business, as well as providing primarily cloud-based managed services and systems management to individuals and very small businesses (1-10 PCs). My primary consulting roles are with Managed Service Providers or in “Genius Bar”-esque improvement positions.

Across my current small client base, I know of one now-premium ODrive1 customer, as well as three free users which have signed up as a direct result of my recommendation. (Related note: Affiliate Referral program?)

I have hesitated to recommend Multcloud as a solution to anyone, despite using it heavily myself, due to the fact that it has some quirks of behavior that can be tricky to navigate for less tech-savvy users. In one case, I have recommended an AWS VM with cloud clients installed, and a couple of Powershell scripts, for a client instead of Multcloud due to concerns over data security. It would be great to have a product, which I would likely already recommend for its core features, be able to provide these value-adds when needed - at additional licensing cost of course!


#11

Something that occurred to me today which might be somewhat of a middle ground: Allow ODrive2 to be run in “service mode” where the sole purpose is to do background cloud-sync stuff.

Currently, Dropbox and OneDrive are “user-based” - meaning you need to link them to a user account on the machine, and run them in the context of that user. Log out - sync stops. You can get around this to some extent with the Headless Linux version of Dropbox (and use NFS/SMB shares to client PCs), but you’re obviously still linked to file sync within a single Dropbox account.

Having something similar for ODrive might be a decent halfway point between “everything is done on the client” and “large data moves done server-side are expensive” where you can use your own (or otherwise accessible) bandwidth and storage to do the sync: DigitalOcean, AWS EC2, etc.


#12

Yes, that kind of thing will certainly be feasible with the server agent version of our software which you can use to control syncing, backups, etc. We’re building it for Windows, Mac, and Linux.

-Jeff


#13

Perfect! Would that software be built specifically for an end user, or would it be built with “service provider” in mind where someone with access to storage and bandwidth can offer that functionality to others? (and, obviously, that would mean it could be a “paid ODrive Official Addion” at some point as well)


#14

Oh, I see what you’re getting at… initially it’s meant to be specific for an end-user in the sense that it’s where you can configure sync and backup folders that you just want to have going continuously but running on some server instance. (One major use case for this, for example, is if you have particular servers that produce a bunch of data that you want to move/collect somewhere else… so you install the server agent and configure it to do what you need it to do). But you could also just use it to offload the bandwidth and other resource usage onto a server in the cloud vs. a local machine.

Your suggested use case is a lot more freeform in what you can do with it. So you can set up your own service and people can submit very specific jobs to do. Once the job finishes, it’s done and doesn’t necessarily preserve the information, especially not as any configuration tied to the “user” that installed the server.

Or maybe it does preserve the job info, but the product has a built-in facility where consumers of this service can create their own specific users on that service to save their particular configuration/jobs.

Interesting ideas for a different but related kind of product offering… thanks! =)


#15

I was thinking more along the lines of “Sign in with ODrive2 (OAuth2?)” so one of many ODrive Service instances running on a machine can “link” to that account, allowing it to process requests on behalf of client apps on that account until they deauth it. I am thinking more of my clients and allowing them to perform cloud moves “natively” while just providing the storage/bandwidth for them.

That is assuming that the service works as a delegate of the client app (running on your desktop/web/etc), and not that you need to configure the “service” as simply a “client that behaves slightly differently” - Sorry if that assumption is incorrect.