Data company SocialCops’ solution for reading and writing data from cloud and local storage is now open-source.
The project, flyio, provides an interface for interacting with data storage directly from R.
Currently flyio supports AWS S3 and Google Cloud Storage, and can read or write tables, rasters, shapefiles, and R objects to a data source from memory. It also enables users to specify which function name they want to read or write.
In the future, SocialCops plans on making flyio functions the default read and write functions for any format in R, the company explained.
SocialCops created flyio because its data sizes grew too large to keep a local copy of it. It needed a way to directly interact with cloud-stored data without disrupting current workflows. There were a few options provided by AWS and GCP, but none of the solutions provided a single way of inputting and outputting data.
“As the data science world is moving towards cloud computing and storage, a bunch of tools and support have come up. With Amazon Web Service (AWS) and Google Cloud Platform (GCP) leading the development and innovation in this space, the battle between their two cloud storage providers, Amazon Simple Storage Service (Amazon S3) and Google Cloud Storage (GCS), continues. We wanted to try out both, while still keeping the option for reading and writing files from our local systems too,” SocialCops wrote in a post.
