All posts (Page 118 of 131)
Despite the hype around “big data”, a more immediate problem facing many scientific analyses is that large-scale databases must be assembled from a collection of small independent and heterogeneous fragments – the outputs of many and isolated scientific studies conducted around the globe.
Collecting and compiling these fragments is challenging at both political and technical levels. The political challenge is to manage the carrots and sticks needed to promote sharing of data within the scientific community. The politics of data sharing have been the primary focus for debate over the last 5 years, but now that many journals and funding agencies are requiring data to be archived at the time of publication, the availability of these data fragments is increasing. But little progress has been made on the technical challenge: how can you combine a collection of independent fragments, each with its own peculiarities, into a single quality database?
...There are many different databases. The most familiar are row-column SQL databases like MySQL, SQLite, or PostgreSQL. Another type of database is the key-value store, which as a concept is very simple: you save a value specified by a key, and you can retrieve a value by its key. One more type is the document database, which instead of storing rows and columns, stores blobs of text or even binary files. The key-value and document types fall under the NoSQL umbrella. As there are mature R clients for many SQL databases, and dplyr is a great generic interface to SQL backends (see dplyr vignettes for an intro), we won’t delve into SQL clients here....
There are two things that make R such a wonderful programming environment - the vast number of packages to access, process and interpret data, and the enthusiastic individuals and subcommunities (of which rOpenSci is a great example). One, of course, flows from the other: R programmers write R packages to provide language users with more features, which makes everyone’s jobs easier and (hopefully!) attracts more users and more contributions.
But what if you have an idea, or a need, but not the time or confidence to write a package for it? I can’t speak for this blog’s readers, but I’ve been writing R for about two years and it took a good long while before I felt comfortable contributing upstream to CRAN. Or, what if you do have the time, and do have the confidence, but want to spend that time well, on things that you know other people will find useful, and don’t know what that is?
...rOpenSci specializes in creating R libraries for accessing data resources on the web from R. Most times you request data from the web in R with our packages, you should have no problem. However, you evenutally will run into problems. In addition, there are advanced things you can do modifying requests to web resources that fall in the advanced stuff category.
Underlying almost all of our packages are requests to web resources served over the http protocol via curl. curl is a command line tool and library for transferring data with URL syntax, supporting (lots of protocols) . curl has many options that you may not know about.
Key to the success of rOpenSci is our community and we want to hear more regularly from our members, and foster new interactions among the group. In addition, community calls are a way for us to give important updates, and get feedback on them.
We tentatively plan on doing community calls once per month. The format of rOpenSci community calls could be of various types. We could have community members show off software they’ve been working on, or users demo use cases. Instead, we could focus more on conversations. For this first call, we’ll be doing a combination of demonstration and discussion. We would like to experiment with the call format over the next few months before we decide on one or more approaches that work best.
...
