This blog was first published on the Filecoin Station blog.
SPARK is checking whether public content stored on Filecoin can be retrieved. To do so, we need to find out which Filecoin deals store data that’s expected to be publicly available.
Filecoin was designed to store all kinds of data, but not all of it is meant to be publicly retrievable. For these “private data” deals, it’s up to the client and the Storage Provider to agree on how the client can access the stored data. Such an agreement happens off-chain.
On the other side of the spectrum is the community program called Filecoin Plus for Large Datasets, often abbreviated as FIL+ LDN. This program aims to incentivise the storage of public open datasets on Filecoin, such as measurements produced by scientific experiments. There is a clear expectation that content stored through FIL+ LDN should be readily retrievable on the network and this can be regularly verified (quoted from current scope in FIL+ LDN docs).
While FIL+ LDN does not cover all publicly retrievable data, it gives us a great start.
How can we find all FIL+ LDN deals to choose some of them to check? There are three steps in this process:
<aside> 💡 You can find our implementation of this sampling algorithm on GitHub at https://github.com/filecoin-station/fil-deal-ingester/.
</aside>
Storage deals are managed by the built-in Storage Market Actor. The RPC API method Filecoin.StateMarketDeals returns a list of all deals created since the Filecoin Mainnet genesis. As you can imagine, it’s a lot of data - more than 20 GB in April 2024 - and the size is steadily growing as more deals are created over time. As a result, most RPC API providers have disabled access to this RPC method.