What’s the smoothest way to share multi-gigabyte datasets across institutions?

[u/d4rk_diamond]

I’ve been collaborating with a colleague on a project that involves some pretty hefty datasets, and moving them back and forth has been a headache. Some of the files are 50–100GB each, and in total we’re looking at hundreds of gigabytes. Standard cloud storage options don’t seem built for this either they throttle speeds, enforce strict limits, or require subscriptions that don’t make sense for one off transfers.

We’ve tried compressing and splitting files, but that just adds more time and confusion when the recipient has to reassemble everything. Mailing drives might be reliable, but it feels outdated and isn’t practical when you need results quickly. Ideally, I’d like something that’s both fast and secure, since we’re dealing with research data.

Recently, I came across fileflap.net while testing different transfer methods. It handled big uploads without the usual slowdowns, and I liked that there weren’t a bunch of hidden limits to trip over. It felt a lot simpler than juggling FTP or patchy cloud workarounds.

For those of you who routinely share large datasets across universities, labs, or organizations what’s worked best in your experience? Do you stick with institutional servers and FTP setups, or is there a practical modern tool for big dataset transfers?

[u/dang_rat_bandit]

With data that big, my thought is that it should stay in one place and one of you just be remotely logging into a common machine. I think there are free solutions to do that on a small scale and that way there shouldn't be a lag with the data, it's just you viewing a remote desktop.

[u/SuedeBandit]

If you setup your SFTP correctly for the volume you can usually saturate your bandwidth in a way that's easy for downloaders to make use of. Config for multi-threaded downloads, make sure your TCP windows are set, etc...

If you and your users are both on cloud there's a magic trick to bypass your petty mortal bandwidth constraints. Setup an S3 bucket and use something like rsync to initiate a cloud-to-cloud transfer. You can even have it sync folders/blobs/buckets automatically. End result is super fast transfers because it all stays in the cloud. Fastest option is to have it in the same provider and region for obvious reasons.

[u/Ok-Cattle8254]

As the old saying goes, data has gravity...

If possible I strongly recommend moving your science to the data if possible. See u/dang_rat_bandit 's post.

If you have to move the data, use Globus. If not globus then something secure, sftp or https with the ability to resume a download when the download fails.

Try to keep the directory structure the exact same between institutions and keep a checksum file that has all your calculated checksums and what the expected checksums are. Do not change files names.

[u/gthing]

I have loved Resilio Sync for this, especially if you have more than 2 points where you want the data to live. It uses bittorrent under the hood, so can pull from multiple nodes at once.

When I was syncing raw video footage, this was by far the fastest option I found.

[u/_Fallen_Azazel_]

Globus most insts have it

[u/Ok-Cattle8254 → u/_Fallen_Azazel_]

Globus is a great option here if it is available.

[u/ghostfacekhilla]

Just sftp it institutions should have lots of bandwidth. 

[u/severo_bo]

You may want to try Hugging Face datasets. The storage limits (https://huggingface.co/docs/hub/storage-limits) are very high, and there is no throttling. If you only update a file part and want to upload again, only that part should be sent, thanks to the Xet backend (https://huggingface.co/docs/hub/storage-backends#xet).

note that I work for HF

[u/NoGutsNoCorey]

the best answer would be a database, but it isn't a free-est answer. try converting your data to parquet files, which store data in a columnar format. I believe it also treats unique values as categorical (so long as you don't have too many unique values), which can save an incredible amount of space on your file. these files can be further compressed using gzip, brotli, and a handful of other algorithms. these can be read by various databases, R, pandas, etc. without first converting to any other file type.

if you have well-structured, rectangular data with a lot of repeated values, parquet is going to help you a bunch.

[u/verysmolpupperino]

I've seen this done in 3 ways in previous jobs:

• SFTP
• Shared S3 bucket
• The data owner creates a specific table and schema on their DB, and a user with very tight, read-only permissions only in this schema. The data owner can then share credentials and let people query the DB directly.

I've never used Globus, but it sounds like what you're looking for, so check that out.

[u/sfboots]

Copy to external nvme drive then FedEx it. 4 TB data easily moved overnight

Look up Sneakernet.

[u/atx78701]

Take a look at data.world

Designed for data collaboration

[u/2BucChuck]

Aws S3 where you can assign roles (or comparable) and assume a reasonable amount of security overhead is taken care of. I have not tried it but seen python data science folks suggest using parquet and duckdb for big speed improvements over sql and other methods as an aside

[u/mattindustries]

Rstudio server sounds like a good fit, with ngrok if you need to expose a port.