# DeSci Codex The Collaborative Data Exchange DeSci Codex is a protocol for scientific publishing that aims to weave a truly persistent fabric of the scientific record, where publications and all associated material (like data, code, visual material, etc) can be addressed with high granularity in a participatory open-state repository. This without depending on any single actor to maintain it, allowing our collective knowledge to be securely stored and openly accessible. Head over to the [Quick start](/readme/quick-start) section to learn what you can do with DeSci Codex. {% hint style="success" %} If you have questions or want to discuss the protocol from a practical or theoretical perspective, come into the [Discord](https://discord.gg/A5P9fgB5Cf)! This project is a community effort, and your thoughts and opinions will help shape its future. {% endhint %} # Quick start Quickly discover what Codex is and what it has to offer Welcome to DeSci Codex! This guide will help you get started based on what you want to do. ## Discovering Content Want to explore research published on Codex? ### Gateway Applications * Discover published research in the [DeSci Nodes](https://nodes.desci.com) application * Explore the raw publications using the [dPID Browser](https://dpid.org/browse) ### Direct Protocol Access * Programmatically explore content using the [dPID resolver API](https://dpid.org/api-docs/) ## Publishing Content Ready to publish your manuscript, data, and code? ### Getting Started * Publish your research to Codex using the [DeSci Nodes](https://nodes.desci.com) application ### Advanced Publishing * Programmatic publishing with typescript/javascript library [nodes-lib](https://www.npmjs.com/package/@desci-labs/nodes-lib) ([library docs](https://desci-labs.github.io/nodes/)) * Learn how to [self-host your publication data](/protocol-operation/data-storage) ## Contribute to the network Want to help maintain the network? ### Node Operations * Run your own [Codex Node](/protocol-operation/codex-node) to support the network * Run your own [dPID resolver](https://github.com/desci-labs/dpid-resolver) ## Building Integrations Want to build on top of Codex? ### Integrations * Build a custom gateway using the [dPID resolver API](https://dpid.org/api-docs/) * Create low-level protocol integrations using [codex-lib](https://www.npmjs.com/package/@desci-labs/desci-codex-lib) * Build a custom DeSci Nodes integration using [nodes-lib](https://www.npmjs.com/package/@desci-labs/nodes-lib) ## Next Steps Choose your path above and dive deeper into the specific documentation. You are always welcome to [join our Discord](https://discord.gg/A5P9fgB5Cf) and connect with the community. This is the place to get help along the way! # Overview High-level information about the protocol DeSci Codex defines an interactive data graph where anyone, can freely collaborate on maintaining the scientific record. Given some base properties of the underlying implementation, the goal is to work together globally, without barriers, in weaving the permanent fabric of our collective scientific progress. Every thread of which is deterministically resolvable and its existence independent of any one individual person or organization. The network collectively contributes to safekeeping our accumulated scientific knowledge. Anyone can create user-facing applications for interfacing with this open-data-network, which is only defining the rules of interaction. Such applications are called gateways, and fill the role of user-facing, application-specific interactions with the underlying graph. We strive for decentralization and diversity in gateways, data pinning, HTTP resolvers, analytics services, scientific collaboration platforms, etc. This will ultimately maximize accessibility, mitigate censorship, guarantee data integrity, reduce operating costs, guarantee long-term preservation, increase democratic participation, reduce gatekeeping, achieve radical transparency, and empower individual researchers. # Design goals These are the properties that the protocol strives to achieve ## Verifiable ownership By controlling the credentials that control the state of research outputs and other contributions, creators own all of their content without reliance on an intermediary. This approach, coupled with robust rights retention enacted through licensing, enables researchers to truly own and control their data. ## Open credit Science is built on collaboration, the protocol must enable collaborative contribution opportunities and a robust and granular credit model. The scientific record should not be composed of static information, but be a living data graph of incremental improvements. Any contribution to science should be creditable, not just the authorship of articles. Proving who did what, and when, needs to be possible for all contributions at the most granular level possible. Peer review, reproducibility checks, metadata cataloging, et cetera, should all be creditable. Knowing who did what when is instrumental in building a robust and fair credit system, so the “version of record” can become a “record of versions”. ## Network participation For the protocol to be truly successful in the long run, it must be useful and extensible by a large, diverse community, and easy to participate in as a network operator. The knowledge of humanity should be able to far outlast any individual and company. This requires a new architecture which is fundamentally open and P2P. The protocol is designed to power a fully open and participatory infrastructure, where anyone can take part in storing copies of open data over a distributed network. ## FAIR principles The FAIR principles stand for “Findable, Accessible, Interoperable, and Reusable”. FAIR infrastructure requires unique and deterministically resolvable identifiers for all digital objects with machine-readable metadata. All properties of the scientific record should be both machine- and human-readable, including publications, their components, and their metadata. ## No Vendor lock-in The scientific record is our collective property, and interacting with it should not be dependent on the continued collaboration of any single service provider. There needs to be separation between data and service provider layer, thus preventing “vendor lock-in” or capture by any single entity. ## Data integrity The scientific record needs to be persistently addressable; including links that continue to point to the same content to which they were once made, and never cease to function. The technology now exists to protect from link rot and eliminate content drift from the scientific record. While publications can evolve, links made to a specific version should still resolve to that specific content. This implies that any update is manifested as the appending of a new version - not overwriting the old version - and that no content can ever be truly removed. For data that cannot be shared openly, publications can still expose the fingerprint of the data. This allows regulatory access through formal requests, to enable complying with legislation. # Codex Nodes Run your own node to help support the network ## What is a Codex Node? A Codex Node is your contribution to preserving humanity's scientific knowledge. By running a node, you become part of a distributed network that ensures research publications, data, and code remain accessible forever. With this resilience, Codex can continue to serve content regardless of what happens to any individual institution or service. ## How Does It Work? Your Codex Node consists of two complementary services: ### 1. **Codex service** Stores and serves the publication metadata and (optionally) publication artifacts like manuscripts, datasets, code, figures, etc. This information is fetched from other peers over IPFS, after which your node makes it available to other peers in the network. So when researchers publish their work, it gets distributed across nodes worldwide, including yours! ### 2. **Ceramic service** Participates in peer-to-peer network gossip to discover and verify new publications, as well as updates to existing publications. This automatic content discovery is what feeds the Codex service information about what new content it needs to find and help distribute. The ceramic node is also responsible for validating cryptographic signatures from the author, and tracking the versioning of each publication. These services run as containers, and together they ensure that both content and context are preserved permanently. ## Why Run a Node? By running a Codex Node, you: * **Preserve Science**: Help ensure critical research remains accessible to future generations * **Support Open Access**: Enable researchers worldwide to access scientific knowledge without barriers * **Increase Network Resilience**: Every additional node makes the network stronger and more resistant to failure * **Contribute to Decentralization**: Reduce dependency on single institutions or companies for preserving knowledge ## Requirements Running a Codex Node is straightforward and requires: * **Disk Space**: At least 100GB available (more is better ) * **Internet**: Stable connection with reasonable bandwidth for syncing content * **Docker**: The node runs as containerized services for easy deployment * Of course, it is also possible to run a Codex Node in a cloud setting as either a VM or in Kubernetes, or as regular system services on your machine. * **System**: Works on Linux, macOS, or Windows with WSL2 ## Getting Started Setting up a Codex Node involves running two Docker containers that work together to preserve and serve scientific content. For detailed setup instructions, see the Codex Node documentation in the [DeSci Codex](https://github.com/desci-labs/desci-codex) repository. ### What Happens Next? Once running, your node will: * Automatically discover and sync existing publications * Vigilantly listen to network gossip and sync future publications * Redistribute content over IPFS * Participate in the network's consensus mechanisms * Help verify the integrity of stored data ## Join the Community Running a node makes you part of a global community working to democratize access to scientific knowledge. Connect with other node operators in the [Discord Community](https://discord.gg/A5P9fgB5Cf), this is the place to get help and share experiences *** Every node matters. By running a Codex Node, you're not just storing data. You are contributing to the safekeeping of a decentralised scientific record for a long time to come. # dPID resolver HTTP resolver for persistent identifiers While the protocol already has globally unique, versionable, persistent identifiers for publications, they are quite verbose, being based on content hashing. The dPID system allows minting human-readable identifiers, which is the main way to reference publications on Codex. The dPID resolver acts as a HTTP bridge to content resolvable in the protocol, allowing programmatic exploration and resolution of publication data over an user-friendly API. [dpid.org](https://dpid.org) is the canonical resolver which is maintained by DeSci Labs, but you can easily host your own. Visit the [browse page](https://dpid.org/browse) to explore content published on Codex and see how a dPID can be used to granularly reference any artifact in a publication. ## Implementation The dPID registry is implemented in a decentralised fashion as a smart contract, which permanently maps a human-readable PID to a Codex reference. This contract is open for anyone to interact with, the protocol data is open, and the [dPID resolver](https://github.com/desci-labs/dpid-resolver) is free, open source software. # Gateway providers User-facing applications for interaction with the protocol The protocol itself isn't meant for direct end-user interaction, but to act as the underlying storage and indexing solution that enable them. A gateway is any type of application allowing end users to interact with the underlying protocol. A gateway can freely choose what data from the protocol to surface. This means that a gateway can decide to skip supporting annotations and attestations and just support a reader experience, or go the other way and be a fully-fledged command center for protocol interaction. The same applies for attestations, does the gateway only show attestations from verified, credible institutions, or any made by unknown community members? Does it show annotations only made by researchers with a verified ORCiD account? There are no right or wrong answers; different use cases may demand different views into the data of the protocol. The pioneering gateway is the [DeSci Nodes](https://nodes.desci.com) platform, providing a user-friendly publication experience, exploration of publications, a journal system, and related functionality. Notable also is the [Insight Journal](https://insight-journal.org/), an innovative journal using Codex to publish and serve publications in an interactive format. {% hint style="info" %} Access to and participation with the underlying protocol will always remain free and uncensored, gateways can be proprietary or freemium services serving high-volume use cases. This unlocks the creation an economy built around interactions with the scientific record, without leading to data silos, paywalls, or link rot. {% endhint %} ## Example future gateways Here are some example gateways that would provide valuable ways of interacting with the protocol. It's likely that several use cases are woven together into a homogenous interface, and that some gateways are specialized for large organizations, but it aims to paint a picture of what rich ecosystem of supporting services would be useful and possible on the protocol. ### Authoring experience Provide a way for a researcher to iteratively build research objects and publish their results. [DeSci Nodes](https://nodes.desci.com/) is such a gateway. ### Data stewardship portal A way for data stewards to find research objects where the author needs help organizing data or code in a publication, or to define correct metadata for their components. ### Attestation manager A portal for organizations to manage claims and requests for attestations and, track application processes and schedule follow-ups. ### Organization manager Allow organizations to keep track of members and their contributions in the protocol, allowing sponsorship of data storage and similar institutional services. ### Reference manager Make collections of interesting works to use for citations, potential collaboration, data or code re-use, or simply as a reading list. ### Graph visualization Graphical representations of the protocol graph, allowing unprecedented exploration in the provenance of scientific discovery. ### Peer-review systems Application for managing peer review flows for journals, authors, and reviewers. This enables a transparent peer-review process, but it could be anchored in an unpublished state for some type of retroactive reveal scheme. ### Funding tracker A portal where funders can track the ongoing progress of research they have funded, from following the progress to evaluating the resulting impact of the funded publications. ### Metrics portal Compiling statistics, impact measures, charts, research field growth, or any number of exciting properties of the open data repository. ### Journals Running a journal is immensely less expensive and complicated when the publications with artifacts are openly available and persistently addressable in the protocol. Providing the services necessary to build and maintain journals is a valuable activity. ### Data pinning trackers Tracking which parts of the graph or research publications are well pinned in the IPFS [DHT](https://docs.ipfs.tech/concepts/dht/), and where community efforts are needed to ensure future stability. ### Metadata services Allowing translation of metadata formats to be used when interfacing between different systems. Additionally, it could allow powerful queries over a metadata index, like searching by schema or joining metadata files with their community suggestions. # Data storage Ensure longevity of protocol data Maintenance of the protocol itself relies on the community nodes pinning and indexing the protocol data, which include publication metadata and versioning history. The data entities in the protocol are designed to minimize the storage complexity of this participation. It is important to point out this does not include the data DAG, or file structure, associated with the publication, because it is orders of magnitude larger than the metadata and cannot be naively included simply from cost of storage. One of the core strengths of Codex is that it has separated the problems of metadata and discoverability from storage and delivery of the much larger files included in a publication. That makes it possible for multiple solutions for data hosting to co-exist, while still being resolvable through Codex and the [dPID](https://dpid.org) persistent identifier system. Platforms like [DeSci Nodes](https://nodes.desci.com) provide a generous free tier for pinning publication data to IPFS, but it is also possible for end users or organisations to self-host that data. From the distributed nature of IPFS, and content based addressing, it does not matter where the data happens to be physically hosted for Codex resolution to work. ## Self-hosting publication data Codex natively supports resolving references to IPFS data by CID. This means it's not required to upload files to the platform where the publication is made, but instead the metadata can define a directory or file as an "external CID". This will then be resolved over the IPFS network. If you publish using [DeSci Nodes](https://nodes.desci.com), the interface natively support adding a component to the publication as an external CID. This is the right choice for including data that you don't want, or need, DeSci Labs to host for you. {% hint style="info" %} If you or your organisation want to self-host data on IPFS, the Kubo client is the most stable and mature. You can refer to the [IPFS documentation](https://docs.ipfs.tech/how-to/) for instructions how to configure Kubo. The IPFS Desktop application might be suitable for a personal machine, but as data is only available when the node is online it's not a good way to ensure data availability when others try to resolve the content of a publication. What you are looking for is likely to configure a highly available server that runs Kubo. Instructions for that can be found in the [IPFS command-line quick start guide](https://docs.ipfs.tech/how-to/command-line-quick-start/), and if you need help you can always reach out in the [community Discord](https://discord.gg/A5P9fgB5Cf). {% endhint %} ## Community DAG replication There is ongoing work in expanding the functionality of the Codex Node to allow for community-backed storage of publication data. The vision is that the Codex network can act as a collaborative pinning network, where participants can choose to "donate" a certain amount of disk space which will be used to pin data from publications. This would reduce the reliance on centralised hosting providers, at least for smaller datasets. ## Storage deduplication There are more advantages in using content addressed storage than location independence, it's also quite efficient in deduplicating stored data. If two research objects use the same large file, a pinning provider only needs to store it once, because if it has the same hash, it *is* the same data. This optimization happens automatically when pinning data with IPFS. ## Data longevity This begs the question: how can the protocol facilitate long-term safekeeping without a formal guarantee of data availability? The answer is diversity in active storage providers, and being able to choose level of persistence guarantees by potentially combining several of them. Not being reliant on a single organization for storage is great for long term survivability, and from content addressed storage we can have deterministic resolution even if the data location changes. An point worth iterating is that the protocol is open by design to different solutions to this data storage problem, and tries to separate that from the maintenance of the contextual data. This is a good thing for end-users because there can be continuous development in this area, with a competitive economy around data services. Some could provide hot storage for frequently accessed resources, others very competitive archival storage rates by interfacing with storage protocols like [Filecoin](https://filecoin.io/) or [Arweave](https://www.arweave.org/). Both of these services are very hard to get right for smaller institutions that aren't mainly in the business of running compute and storage infrastructure. ## Compute-over-data capabilities A publication with results based on large datasets are naturally difficult to reproduce because of the challenges in moving large amounts of data. Content addressed storage opens up for distributed compute over data technologies like [Bacalhau](https://www.bacalhau.org/). In this case, a storage provider could also run Bacalhau nodes, allowing users to send compute jobs to reproduce results to execute on the same site where the data is stored, on capable machines. This efficiently avoids costs associated with data egress and maintaining their own compute infrastructure. # Introduction to Sidetree A primer on the required capabilities of implementing technologies {% hint style="warning" %} This section introduces some underlying base properties necessary for practically implementing the protocol. This is not required to get the gist of , so it can be safely skipped if a general overview is the goal of your read. {% endhint %} The protocol is a graph abstraction, but there are certain base properties that need to be supported by the implementation for the requirements laid out in the [Design goals](/readme/design-goals) to hold. These properties partially overlap with the definitions made in the [Sidetree Protocol](https://identity.foundation/sidetree/spec), which is practical as a common language in this specification. Sidetree is a higher order protocol, i.e., lays out shared definitions and necessary properties for an implementing protocol to achieve certain outcomes. It is originally intended for decentralized public key infrastructure, but one can generalize the concept and apply the pattern for other types of documents. The basics of Sidetree is that nodes in a network collaborate to keep track of sequential updates to documents made by end users, in a trustless fashion. An end user who wants to update their document announces this to a node in the network, which verifies that the cryptographic signature of the update corresponds to the one who created the original document. If this is the case, the node includes a persistent identifier of the update message in a tree that is posted to a decentralized anchoring layer, like a blockchain. What's important here is that each single transaction is recorded in the anchoring layer in large batches, enabling great reduction in throughput and cost. The nodes in the network communicate about new events they have received. As nodes learn about a new update, they can verify that it is indeed authentic, and check a proof of inclusion in the public ledger. By sharing and verifying document updates in this way, the network reaches consensus of what is the latest state of a document without having to establish trust. ## Central properties In the Sidetree summary above, there are several essential properties that the DeSci Codex leverages in order to meet its design goals. ### Verifiable authorship This means that each and every user interaction within the protocol needs to be cryptographically signed to establish provenance. Without this property, one could never be sure if an action was really taken by the assumed person or that data hasn't been tampered with. ### Anchoring scheme Time and ordering are both difficult problems in decentralized systems, and to reach global consensus we require protocol nodes to anchor the received events in a public, immutable, append-only anchoring layer. It is not scalable for every node to hold the entire state of history, so it needs to be possible to validate inclusion in constant time and space. This is generally achieved with just posting the root hash of a Merkle tree and just announces a small [Merkle proof](https://computersciencewiki.org/index.php/Merkle_proof) of the inclusion which is easy to verify. All in all, the main takeaways from the anchoring scheme are: 1. Avoiding rampant storage swell on the anchoring layer through a minimal and efficient method of proving inclusion. 2. Avoiding direct user interaction with the anchoring layer, which is in general much slower than any Sidetree based protocols. ### Versioning As every author-verified update request to the state of a document is cryptographically tied to the previous version and independently anchored, we have an ordered history of updates to each document. This is very important for being able to persistently identify the state of a document at a certain point in time, which is a key attribute necessary to reach our design goals. This is what allows us to maintain a record of versions, compared to the current singular version of record that is the industry standard. ## Adapted terminology Sidetree uses the DID suffix for document identifiers and a value called the commitment for operations. In the protocol, we will refer to them as identifiers and commits. Additionally, we only consider `Create` and `Update` operations as `Recover` and `Deactivate` does not make sense in this context. We will also talk about different entities, which can be considered schemas of data nodes in the graph. This is analogous to a Sidetree node handling different types of documents, and validates the schemas of each type. ## Note on indexing Partly simplified but otherwise accurate, the Sidetree spec is only concerned with validation and consensus and leaves no details on efficient indexing and querying for data in documents. This is an implementation detail which is solved under different constraints depending on the implementing protocol, but is a significant consideration when choosing the underlying technology to back the DeSci Codex. # Entities High-level overview and charts explaining the grand scheme of things The protocol models the scientific records and all contributions to it as a graph. The reason this representation is chosen is that it enables clear and explicit relations between entries, like the connection between a publication and its author, or a scientific reference between two different publications. Each node in this graph is cryptographically signed by the author of the data, meaning we have verifiable provenance for every contribution ever made. The schemas, or models, of a node is called an entity, and a user-created entry based off that is called a node. The users of the protocol participate in curating metadata, and populate and enrich research objects. All parts of the graph are user contributions, and the range of available entities can also be extended by the community.

An example of a graph of independent authors, their publications, and the relations between the different types of entities.

{% hint style="info" %} This graph is an initial overview to give some basic understanding of author ownership, collaboration, and the graph concepts that will be mentioned going forward. This may be a new and confusing way to represent things in this space; the main takeaway should simply be thinking about the protocol as a graph. {% endhint %} ## Field assumptions Some values are assumed to be present in every instance of every entity type as it is created in the protocol, as they are necessary for verifiable authorship and deterministic resolution:
FieldTypeDescription
ownerDIDUnique identifier of the entity creator
idIDUnique identifier of this node
commitCommitUnique identifier of the particular version
## Arguments for fragmentation Compared to the previous version of the protocol, where all data was kept inside a single manifest file on IPFS, the new version has several entity types which instead hold references to one another to shape the graph. One could argue that this increases complexity because the same data is spread out in different places. This is correct in one sense, but there are multiple arguments to why this design is stronger. ### Extensibility This fragmentation enables anyone to create a new entity that adds relational information or functionality and connecting it to the data layer with references, or edges, in the graph analogy. This allows the community to retroactively expand context without requiring the original entry to be changed, which is particularly powerful because we can continue to enrich information around unattended entries in the scientific record. An additional benefit is that there does not need to be agreement on the schema of the manifest. Some independent set of actors can just join forces in creating a new entity and index that, regardless of the opinion of the broader community. Maybe it is then proven to be useful, in which case it is easy for the rest of the community to start indexing that information as well. ### Ease of indexing Lowering the bar of participation and building supporting services is important for longevity of the protocol. As a concrete example, the [Research field](/data-layer-definition/entities/relational-information/research-field) entity allows the identifiers of publications in different fields by looking at a very small set of data. Being able to do this without a separate entity would require keeping a database linking every single published manifest with its identifier, and perform a quite computationally intensive search through that data to filter on the author-defined research field. With this dedicated entity, a community member wanting to build a simple website listing links to publications in particular fields will have a much easier time to do so without depending on someone else for the indexing. ### Granular ownership With each contribution to the scientific record being independently associated with its author, it's simpler to credit activities that are significant but aren't adding to the reputation of the author in the classical system of science. This is an important enabler for building incentives for performing supporting services on the scientific record like community review, data stewardship, improving metadata, and a plethora of other tasks. The following sub-pages dig into the functionality of each entity of the data layer. # Profiles User identification and relations Before looking at the profile, which is a protocol entity representing a user's public information, it's important to clarify the difference between this and the user's DID. ## What is a DID? A DID, or Decentralized Identifier, is a new type of identifier that is created, owned, and controlled by the subject of the digital identity, without the need for centralized authorities or intermediaries. DIDs are stored on distributed ledgers or other decentralized networks, ensuring their security and verifiability while giving individuals greater control over their digital identities. A DID is portable in the sense that it can be used in different places; it is not tied to a particular service like the traditional user account. DIDs come in different shapes and forms, known as providers. A DID provider is a scheme for validating and updating a DID, a common implementation of which is variants on Sidetree as discussed in [Introduction to Sidetree](/data-layer-definition/introduction-to-sidetree). Different providers may have different capabilities, like allowing the user to use multiple keys, control the identifier just using a blockchain account, perform key recovery and rotation, and similar functionality. For the context of the protocol specification, we can think of it simply as an actor-controlled cryptographic key, because that is the basic property that's required to validate user control over entity authorship. Visually, a DID looks something like this: ``` did:pkh:0xb9c5714089478a327f09197987f16f9e5d936e8a ``` ## Profile Since DIDs doesn't always follow a certain structure, we can't generally enter protocol specific information. Even if we could do so, creating references between other protocol entities and the profile is the main benefit of having a dedicated representation of user data in the protocol. The profile entity fulfills this need for linking user-specific information with a DID. As with all other entities, it is under the exclusive control of the creator DID and cannot be updated by anyone else. Perhaps the most valuable information, at least initially, is other social handles of the user.

A profile entity instance listing the author's real name and relevant social handles

## Schema
FieldTypeDescription
displayNameStringHuman-friendly name of the DID owner
publicKeyStringPublic key of the DID owner
orcidStringORCiD handle
googleScholarStringGoogle Scholar handle
{% hint style="info" %} This selection of fields is just for demonstration purposes, likely handles need to be associated as a generic key-value JSON map to be forwards-compatible. {% endhint %} ## Identity verification The public key field can be used to validate the identity behind the DID and prove its association with the social handles. This mechanism is not part of the protocol per se, but is a useful enough operation that it's worth elaborating on. For a user to enable verification of association for a social handle listed in their profile, they cryptographically sign a message stating this with the private key corresponding to the public key in their profile. This message is then made available by the user on the platform in question. ### Practical example {% hint style="info" %} In this example we use the well-known application [`gpg`](https://www.gnupg.org/), but it can be done in various ways. {% endhint %} A user enables verifying control of a social handle by using the private key corresponding to the public key in their profile to sign a message containing their DID fingerprint: ```sh > gpg -v -o- --clearsign <(echo "did:pkh:0xb9c5714089478a327f09197987f16f9e5d936e8a") -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 did:pkh:0xb9c5714089478a327f09197987f16f9e5d936e8a gpg: RSA/SHA256 signature from: "9860CD32C3A74989 Firstname Lastname me@domain.com>" -----BEGIN PGP SIGNATURE----- iQGzBAEBCAAdFiEEJD2AvUlpXneBH9qPmGDNMsOnSYkFAmU3xbYACgkQmGDNMsOn SYntMwv/ccA3ktC+VFxfm3gC5fKoj6GovXM7v/QMJr9zi3olKDUQ7uJtxxP3y2Eb sdbWb9SUPJoDbdSCAI8xJXBRKljNdHFCus1VaJMUpL0eMIa2PfukVKBL6EbZxl2M 9iTmHovA1+Wxxqk+S5IGRVA/hFAYyblbwAyGFTeIYbbuMejFYymjSii44khKDMSb mWJTlbTg5Ck8nWW4JGi/kgtq3UfhxagHFO8SCZ10K3/FApcb6r7iDsvkbvIPDXSS dKpdMI9drn+I484KHxsJ+gx8CTj5ZbMo94HVaUwx1yeVyqymP6CR9m4wyV9+e8Oz C4eXsIfxdxvTqmD5qblUqmYpy8XcXgQcmq0F7751pCsrznYFQD++xM1ZHvwwqIj7 5E/9RgbE8Ox9IoJS6PBZlqgVbQ2stE5j6fUqqwHbw99oBjzkw6s2hoIJ5Q4eXZqb GYfnN+76m2f0ui/jIhUMZAT0G9Nie4xnWUxt7XfDe7cdYULZP2n6C9GnFRY3MEk0 ibipa7gf =1+R5 -----END PGP SIGNATURE----- ``` The resulting message is then made available on the relevant platform, for example in the user profile. As a third party that wants to verify the identity of this user, first save the user's public key to a file called `key.asc`, and the signed message as `msg.asc`. Then, decrypt the message: ```bash > gpg --decrypt --keyring key.asc msg.asc did:pkh:0xb9c5714089478a327f09197987f16f9e5d936e8a gpg: Signature made Tue Oct 24 15:24:47 2023 CEST gpg: using RSA key 243D80BD49695E77811FDA8F9860CD32C3A74989 gpg: Good signature from "Firstname Lastname " [trust level] ``` With this decryption successful, validating the signature was indeed made by the same key, we only need to check that the attached DID fingerprint matches the one actually controlling the profile. The reason for this is that otherwise, a fraudulent user could list the same public key in their profile and copy the encrypted message to *their* social account. With this DID fingerprint payload, we instantly know this is incorrect. # Research object The main reference to a publication and its version history The core of the protocol is the research object, a container for a publication and its arbitrary associated artifacts with a public and deterministically resolvable version history. Everything else in the protocol exists to enrich the context around the research object, giving access to unparalleled analytics and mechanisms for scientific review, while having to place trust in neither central points of failure nor authority. The main purpose of a research object is to keep a reference to a tree of arbitrary data. This can be thought of as a hierarchy of files just like in your personal cloud storage, but publicly available on IPFS. In the ideal case, this tree would contain a research paper, presentations, code, and the data necessary to explain and reproduce the claims of the publication. The research object is mutable in the sense that it can be updated to add additional files or metadata, but static in the sense that every version has its unique identifier and is independently traceable through the protocol.

A research object entity and a visualization of it's link to the data DAG

## Schema
FieldTypeDescription
titleStringTitle of the publication
manifestCIDInstance of a manifest, initially 0.2.0
metadataCIDJSON representation of arbitrary metadata
### The manifest The `manifest` field holds a CID to a JSON file which contains a pointer to the data DAG as well as information about the research object itself. The manifest has a self-documenting version field, and is initially an instance of [`desci-nodes-0.2.0`](https://github.com/desci-labs/nodes/blob/5028c15c90dc533d257b4d343cbc92b68dda27e5/desci-models/src/ResearchObject.ts#L18). Hence, a research object created using the old manifest format is valid and can easily be identified. The reason for this being the initial default is to maintain backward compatibility, while also enabling contextual information as data entries in the protocol. As the protocol matures and the representation of contextual data therein stabilizes, simplified manifest schemas can be released to eliminate this duplication. The data DAG, known as the `DATA_BUCKET` component in the manifest format, is the root CID of an [UnixFS](https://github.com/ipfs/specs/blob/main/UNIXFS.md) IPLD data structure. This can represent any hierarchy of arbitrary serialized data, comparable to the classic file system abstraction. Here is an example UnixFS DAG, encoding a directory holding three files with name and size: {% code title="Example " fullWidth="false" %} ```json { "Data": { "/": { // unixFS protobuf encoded directory indicator "bytes": "CAE" } }, "Links": [ { "Hash": { "/": "bafybeiggs56o2lfnokepfnhllazq4pgohmdnmsdjrjdbxsyntmq2zlktri" }, "Name": "chan_l6_4x12x24_4x4x8.h5", "Tsize": 10626444 }, { "Hash": { "/": "bafybeibeaampol2yz5xuoxex7dxri6ztqveqrybzfh5obz6jrul5gb4cf4" }, "Name": "chan_l6_4x12x24_6x6x12.h5", "Tsize": 35847020 }, { "Hash": { "/": "bafybeibvt5s7scy6lvu6v5r3w2oiliti326ddtpx3hhtvphxpxpaeoiy2i" }, "Name": "chan_l6_4x12x24_8x8x16.h5", "Tsize": 84960093 } ] } ``` {% endcode %} ### Attached metadata The `metadata` field holds minified JSON, the specific format of which is not strictly defined. This metadata file holds all properties that may be required in particular research fields, certain regulations, by specific journals, or other standards. Most of this information will not be set by hand, but either automatically compiled by the publication gateway or suggested by other actors in the protocol, for the owner of the research object to accept. We will return to this concept of metadata contributions later, in the [Annotation](/data-layer-definition/entities/annotation) chapter. A full summary of the metadata capabilities can be found in [Metadata](/data-layer-definition/metadata). ## Self-sufficiency One could view this entity as the only primitive of the protocol, and all other as extensions. In fact, all other entities could have been created by the community. Additionally, to reach functional parity with the previous generation of dPID resolution (details of which we will get into later) we only need the Research Object entity holding a manifest CID to keep track of the entire record of versions. # Research component Fractional information about the contents of the data DAG of a research object. A component entity provides context to nodes in the data DAG of a research object. This could be to comment on its use, indicate the type of data, give it a descriptive name, and attach arbitrary metadata. The research object author uses instances of these entities to enrich context around important parts of the publication, like a dataset, a paper, or a piece of code. Other actors can also create component instances to enrich the context of a research object, or to include the same CID's as part of their publications. A user-facing gateway can choose how to reconcile these different sources of information, and potentially use other author's components to suggest metadata for similar files.

A visualization of the relationships between research objects, its components, and individual files on IPFS

{% hint style="info" %} Since any actor can create instances of a research component, it's likely that a gateway operator only will show those who are created (or accepted) by the DID that published the research object. {% endhint %} ## Schema
FieldTypeDescription
nameStringDescriptive name of the component
mediaTypeStringMedia Type indicating the type of data
researchObjectIDIDUnique identifier of the target research object
researchObjectVersionCommitUnique identifier of the research object version
dagNodeCIDTarget node inside the data DAG
pathToNodeStringThe unixFS path through the DAG (since a CID could exist in more than one place)
metadataCIDJSON representation of arbitrary metadata
### Media type To aid gateways in picking how to represent data, individual files should have information about file type attached. This information isn't otherwise available, part from the extension if that's included in the UnixFS filename field. There is a [rich variety](https://www.iana.org/assignments/media-types/media-types.xhtml) of media types to pick from, and a gateway can implement a fallback representation based on the [top-level type](https://www.rfc-editor.org/rfc/rfc2046.html#section-3) and map specific viewers depending on the subtype. An example of this would be showing `text/*` as regular text, but specialize the view of `text/csv` data as a table. Since not all file types have a media type, a gateway can also draw conclusions from the file extension. A good example of this are code files in different programming languages. Conversely, not all media types have a singular corresponding file extension, so both sources of information are required to paint a rich picture of the content. ## Motivation of separation from research object Separating component information from the research object creates a specialized dataset which simplifies reverse CID look-ups, like finding which research that use a particular dataset or contain the same PDF's, and similar questions. Consider the diagram at the top of the page, where we can go through component entities and find which research objects have a particular dataset included in their publication. The reason this is not embedded directly in the IPLD DAG is that it would change the CID. This would make it difficult both to trace re-use of data in other research objects and reduce usefulness of IPFS deduplication in nodes, which lead to higher maintenance cost for the network. # Claim A reusable badge to be earned Claims are stand-alone attributes, like a diploma without the recipient's name filled in, or a unique rubber stamp. They can be thought of as badges, where the value is dependent on who issued it. These badges are a way to incentivize and validate contributions to the scientific record, and allow building of credibility by recognition in the community. They are connected to the creator DID, which decides the rules of validation depending on the type of claim.

A stand-alone claim with an associated badge for visualization

{% hint style="info" %} The assignment of a claim is called an [Attestation](/data-layer-definition/entities/attestation), which we will cover in the next chapter. A claim on its own is quite abstract, but the next chapter will likely clear things up. {% endhint %} ## Schema
FieldTypeDescription
titleStringTitle of the claim, e.g. "Reproducible" or "Published in X"
descriptionStringDescription of the attribute it represents and on what grounds it should be granted
badgeIconCIDVisual representation of the claim
protectedBooleanValid only if granted by claim creator
reciprocalBooleanValid only if accepted
### Badge icon An image representation of the claim, which should be in square aspect ratio and on a transparent background. Preferably in SVG format, otherwise PNG in 300×300 pixels. ### Protected & reciprocal status The attestation of a *protected* claim is only valid if created by the same actor who created the claim. In other words, all grants of the claim done by any other actor are invalid. The attestation of a *reciprocal* claim is not valid until it has been accepted by the owner of the recipient entity. It can be considered pending until then. All combinations of these two values are distinctly expressive, as shown below. Do note that these examples are merely illustrative, and could likely be represented in other ways as well.
ProtectedReciprocalMotivating example
✅✅A claim like Research funded by NASA should be both protected and reciprocal.

Protected because no-one other than NASA should be able to issue it validly.
Reciprocal, so the author can consent.
✅❌A claim like Published by Science should only be protected.

Protected because only Science (the journal) can decide what they publish.
Non-reciprocal, because a real-world side effect has taken place either way; the recipient revoking their consent does not make it unpublished.
❌✅A claim like Publicly funded research should just be reciprocal.

Non-protected because there is no central authority that controls it.
Reciprocal, so the author can consent.
❌❌A Good Quality community badge should be neither.

Non-protected because it's meant to be granted by anyone.
Non-reciprocal because there are no negative effects of its reception.
The next chapter includes visual demonstrations of how validation of attestation on the different types of claims is done in practice. # Attestation The grant or assignment process for a badge An attestation expresses the opinion that a certain target fulfills a given claim. To expand on the analogies from the previous section, an attestation corresponds to filling in the name in the diploma, or using the rubber stamp on a document. The value of an attestation depends on both the claim and the issuer, but could represent a wide selection of properties: * Curation by a prestigious journal * Passed a FAIR data review * Result reproducibility verified * Metadata fulfilling a standard * Data and code available These attestations build credibility for actors in the system, not only for publishing original research but for any number of other contributions to the maintenance of the scientific record. This is a way to get verifiable certifications for contributions, which can be used in the real world for securing research funding, qualifying for programs, or securing positions based on verifiable merit. Many publishers and funding agencies are starting to push hard requirements for data availability and open science practices, and the attestation system in the protocol can express the certifications to prove it.

The abstract representation of an attestation being granted to a research object, awarding it a badge of reproducibility.

## Schema
FieldTypeDescription
targetIDIDUnique ID of the target entity
targetVersionCommitSpecific version of the target
claimIDIDUnique ID of the attestation claim
claimVersionCommitSpecific version of the claim
revokedBooleanWhether or not the attestation has been revoked
### Target The attestation can target any protocol entity, including attestations themselves. This is a very flexible primitive which aims to unlock a variety of use-cases by the community that cannot be foreseen in advance. ### Revocation If the attester changes their opinion on the target's fulfillment of the claim requirements, the attestation can be updated to revoke. The update history of the attestation will still show when it was initially assigned, and from which target version it should be considered revoked. Ideally, an [Annotation](/data-layer-definition/entities/annotation) is made to motivate the revocation. The attestation should be considered valid for the target versions in between granting and revoking. ## Separation from claims A common question is why there is a distinction made between attestations and claims. Making the claim a separate entity means that an instance made by a credible organization has a unique identifier together with verifiable provenance binding it to that organization. This identifier can be used to find all attestations made with this claim, and the identity of the claim maintainer to validate the attestations made. The alternative would be that each attestation is unique, and the creator could freely choose a badge icon, title, and description, making it a much harder search problem to figure out authenticity and provenance. An attestation with a dedicated claim is easily verified by looking at the DID's that signed the different entities. ## Validation rules The boolean values `protected` and `reciprocal` on a claim as explained in the previous chapter are difficult to illustrate without the attestation validation rules. Here are a set of diagrams illustrating which attestation pattern and hence validation rules are required for each combination of these properties to be considered valid.

Valid attestation pattern for a claim that is non-protected and non-reciprocal, essentially without constraints.
The claim and the attestation can be made by different actors, and the creator of the target does not need to consent.

Valid attestation pattern for a claim that is protected and non-reciprocal.
This differs to the previous diagram in that the creator of the claim and the attestation must be the same actor.

Valid attestation pattern for a claim that is both protected and reciprocal. Note that the attester is the creator of the claim, and in addition, the creator of the target has mirrored the attestation.

### Requests, issuance, and confirmation For protected claims, an actor should be able to request it to their own work. If we use the mirroring attestation as explained in the diagrams above, depending on the order of creation this can be thought of in two ways. The first, **issue & accept**, starts with an attestation from the creator of a protected claim. If the owner of the target entity consents, they create the corresponding attestation back, closing the circle. In the second case, **request & grant**, the author of a piece of research does an (invalid) attestation of a protected claim to an entity they own. The creator of the claim can get notified, review the request, and if everything checks out they attest the other direction and close the circle. The end result in both scenarios is the same, but there are two ways of getting there, depending on which party initiates the process. ### Alternative consent pattern Instead of mirroring the attestation, one could also imagine a dedicated claim indicating consent which can be used to accept an attestation. The main issue this would solve is that the mirrored attestations are on their own invalid in case the claim is protected, which is a sort of exception to the rules that may be trickier to implement. The drawback of this is that it blocks the mechanism of requesting attestations of protected claims as described in the previous section, so some other process would be necessary to capture that flow.

Alternative way to represent attestation consent, using a dedicated type of claim instead of attestation mirroring.

# Annotation Social commentary mechanism An annotation is a localized comment on another entity, which could represent feedback in the case of a review or author arguments explaining why a research component supports the attestation of a claim. Annotations referencing claims can be useful as motivation before an attestation is made, and providing such arguments contributes to user reputation as the discussion is credited.

Annotation thread directly on a research object

Publication author motivates the Data Available claim, and receives feedback from a reviewer

{% hint style="info" %} It’s likely that a mechanism for fighting spam will be required in gateways, for example only showing annotations from profiles with a verified identity. {% endhint %} ## Schema
FieldTypeDescription
commentStringContextual commentary
researchObjectIDIDResearch object context
researchObjectVersionCommitResearch object version identifier
targetIDIDWhich node the annotation applies to
targetVersionCommitVersion identifier of the same
dagNodeCIDOptionally reference a node in the DAG
pathToNodeStringunixFS path to dagNode (it's not necessarily unique)
locationOnFileStringOptional location specifier on the file in question
claimIDIDOptional mention of a claim
claimVersionCommitClaim version identifier
metadataPayloadStringOptional suggested metadata JSON patch, in the case the target entity has such
### Comment Textual commentary in the shape of feedback, arguments for claims, questions for authors, and similar type of information. ### Path If the annotation target is a research component, the annotation author can further specify the location that the annotation refers to. A standard for the different media types is yet to be set, but here are some illustrative examples:
Media typePath syntax
textLine and column number
appplication/pdfPage index and X/Y percentage coordinates for a selection
videoFile timestamp
application/jsonJSONPath
### Metadata payload The annotation can attach a suggested delta to the metadata. This is useful for suggesting adding, modifying, or removing metadata entries. The suggestion is credited to the annotation author, but it's up to the target owner if they choose to apply the suggestions. Regardless, the suggestion is still available. The exact specification of this format is pending, but likely some representation of a JSON CRDT patch. ### Annotation targets Annotations can technically target any type of protocol entity, but there are some particularly obvious applications. #### Research object The annotation is considered to apply to the entire research object. This can be used to leave feedback or questions for the author. A valid assignment has these fields set:
FieldSet?
researchObjectID✅
researchObjectVersion✅
targetID❌
targetVersion❌
dagNode❌
pathToNode❌
locationOnFile❌
#### Research component The annotation is related to the research object context, but refers to a specific research component. This can be used to leave feedback, but also to submit a metadata delta for the author to consider. A valid assignment has these fields set:
FieldSet?
researchObjectID✅
researchObjectVersion✅
targetID✅
targetVersion✅
dagNode❌ (transitive from component)
pathToNode❌ (transitive from component)
locationOnFileDepends: on the file itself or a particular section
#### DAG node (file in tree) The annotation is related to the research object context, but refers to a specific file in the tree. This can be used to leave feedback or questions on a paper, a piece of code, or part of a dataset. A valid assignment has these fields set:
FieldSet?
researchObjectID✅
researchObjectVersion✅
targetID❌
targetVersion❌
dagNode✅
pathToNode✅
locationOnFileDepends: the file itself or a particular section
{% hint style="warning" %} Do note that metadata is mainly suggested against components, because that holds the authorative metadata file. A gateway could find and suggest component metadata updates from raw DAG node annotations, or choose to consider it community contributed metadata. {% endhint %} #### Annotation (a reply) The annotation is related to the research object and, transitively, the location of the parent annotation. This can be used to reply to feedback or questions. A valid assignment has these fields set: | Field | Set? | | ----------------------- | ---------------------------------------- | | `researchObjectID` | :white\_check\_mark: | | `researchObjectVersion` | :white\_check\_mark: | | `targetID` | :white\_check\_mark: (parent annotation) | | `targetVersion` | :white\_check\_mark: (parent annotation) | | `dagNode` | :x: | | `pathToNode` | :x: | | `locationOnFile` | :x: | ## Public and private workflows In some applications of annotations, it may make sense for the content no to be available until a later point in time. Examples of this could be anonymous peer review, non-public conversation, et cetera. In this case, created annotations could be anchored according to the Sidetree implementation, but its content not made publicly available. That allows the content to be revealed later, with proofs of who authored what and when it was created. ## Visibility Not all authors may want public commentary on their works, in which case gateways could decide to only show annotations made on entities where the creator has selected to accept this. This is a gateway implementation detail, as the underlying data graph is permissionless and has no notion of blocking data contribution from authors. # Relational information Additional entities signalling relationships between others The entities in this section act as links or relations between other entities, adding meaningful edges to the graph for analytics and credit tracking. Note that this selection of entities do not capture the entire metadata capability of the system, only to model the graph relations for a subset of properties. The entities[Research object](/data-layer-definition/entities/research-object) and [Research component](/data-layer-definition/entities/research-component) can both hold uncapped collections of arbitrary metadata, and the [Attestation](/data-layer-definition/entities/attestation) captures community suggestions for said metadata and is technically applicable to any type of entity. # Research field Indication that a research object is part of a particular field To aid navigation and indexing, actors can create edges indicating association with particular fields of research. Through these relations, indexers can allow filtering research that is part of both neuroscience and economics, for example.

Research field relations modelling groups of research objects

{% hint style="info" %} Since any actor can create these relations, a gateway operator may choose to only show such indications created by the publication author or an otherwise trustworthy actor. {% endhint %} ## Schema
FieldTypeDescription
nameStringName of the research field
descriptionStringDescription of the same
## Governance To avoid multiple research field instances intending to capture the same, and hence fragmenting the graph, ideally the community adhere to a standardized list. This prevents the same research field existing with different descriptions, which in the end makes indexing more difficult and is confusing to end users. The DeSci Foundation aims to maintain such a set of research field instances based on the [FAIRsharing Subject Ontology](https://www.ebi.ac.uk/ols/ontologies/srao). This is recommended to use as a whitelist for gateway operators when displaying research fields or allowing the creation of such relations. # Contributor Directional link between research object and profile, indicating a contribution. Relates a profile to a research object and indicates a contribution to the value of the publication.

An author-made contributor relation pointing to another user's profile

{% hint style="info" %} Since any actor can create these relations, a gateway operator may choose to only show such indications created by the publication author or an otherwise trustworthy actor. {% endhint %} ## Schema
FieldTypeDescription
roleStringRole filled by the contributor
researchObjectIDIDThe research object contributed to
researchObjectVersionCommitVersion when this relation was created
contributorIDIDThe profile of the contributor
fallbackInfoStringJSON data with social handles, in case contributor does not have a profile
revokedBooleanIndicate if this contributor relation has been revoked.
### Roles To the largest possible extent, the contributor's role should be set according to [CRediT](https://credit.niso.org/), the Contributor Roles Taxonomy. This is to reduce the number of different values, making indexing easier. The linked page has definitions for each of the 14 contributor roles, with descriptions on what classifies as that type of contribution. If there is the need to use some other role identifier, that is still possible. ### Contributors and profiles If the contributor has created a profile in the protocol, it's advantageous to refer directly to that because the contributor can update their social handles and other information. Additionally, it makes it easier to find their own work and other contributions if there is an explicit link. Otherwise, the `fallbackInfo` field allows a key-value map in JSON format for listing the contributor's handles on services like ORCiD, Google Scholar, etc. ### Revocation Since entity instances can be updated, but not deleted, the way to recall a contributor indication is by updating this field to `true`. Gateways should respect this and mark it as revoked from the version where it happened. # Reference Directional link between two publications indicating scientific dependence Relates a research object with another, indicating that the result depend on or is related to the target publication. The exact meaning of this relation depends on the author intention, in the same way as a reference in a classical paper does. Having dedicated entities for references, instead of text based equivalents in the paper, enables querying the scientific record based on direct or transient reference relations.

Reference relations, showing transient references between three publications

{% hint style="info" %} Since any actor can create these relations, a gateway operator may choose to only show such indications created by the publication author or an otherwise trustworthy actor. {% endhint %} ## Schema
FieldTypeDescription
fromIDIDResearch object at the source of the reference
fromVersionCommitVersion indicator of the same
toIDIDResearch object at the target of the reference
toVersionCommitVersion indicator of the same
revokedBooleanRevokation status
### Revocation Since entity instances can be updated, but not deleted, the way to recall a contributor indication is by updating this field to `true`. Gateways should respect this and mark it as revoked from the version where it happened. # Metadata A rich, living repository of arbitrary metadata The different protocol entities, or types of nodes in the graph, express the main relationships necessary to identify and find the context of research publications. This does not mean that this graph limits what metadata can exist for a publication and its surrounding artifacts. On the contrary, there is likely no more capable system for collaboration on unbound metadata in existence. This is because the protocol allows unrestricted metadata to be associated with both graph nodes and arbitrary IPFS CID's, curated by the community and freely available under persistent identifiers. ## Author controlled metadata files Some entities have a `metadata` field, holding the CID of a metadata file scoped over the entity. There is no technical limitation to the format of the content, nor preventing following multiple schemas within the file, separated by a delimiter. The protocol is agnostic to the shape of the data, it only knows about a generic file reference. The user who created the node can update the information in this file freely, but all historical versions are preserved as part of the protocol. A [Research object](/data-layer-definition/entities/research-object) have this file associated, and is under direct control of the author. The author can also create a [Research component](/data-layer-definition/entities/research-component) for particularly interesting files in the research object, which holds the same type of file. Both research objects and components can be targeted with annotations holding metadata payloads, deltas that suggest addition, update, or removal of pieces of the metadata. The author can accept these to patch target nodes, publish it in a new version together with metadata contributor relations to the annotators:

An annotation suggesting a metadata patch on a research component made by the publication author.

Since components and the associated metadata is bound to a particular CID, it's easily found for another author making a publication using the same dataset. A publishing gateway can find this component and propose the information from that component to the author:

Thanks to content addressing, metadata from shared datasets can be reused and suggested to authors by the publishing gateway.

## Community curation Annotations can be used to expand and enrich metadata in the context of research objects and components, but the annotation entity is very expressive on its own. They can attach metadata payloads to arbitrary nodes in the data DAG of any research object, which means every single file and directory included in a publication could have its own set of metadata. The annotations can even be localized inside the files, contributing metadata to a particular figure inside a paper. Even if the author does not acknowledge and merge the suggestions, the metadata edits are still there as standalone nodes and can hence be indexed and displayed as community contributed suggestions. ## Fluid metadata vision The annotation entity supports another scenario where there are no authoritative metadata definitions at all. Annotations can attach payloads in the form of [JSON CRDT patches](https://github.com/streamich/json-joy), a [conflict-free replicated datatype](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type) implementation, which together would yield the current state of metadata. This would remove the notion of a static metadata definition under central control, and instead work by deterministically reducing a collection of metadata fragments into a cohesive document.

JSON CRDT patch based metadata curation, where there would be no metadata directly on any entity, but only compiled from annotation suggestions.

There are some open questions left before this idea can become reality, mainly regarding indexing stability, deterministic history resolution, and the effect of bad actors. ## Indexing system All metadata files are stored on IPFS because the size is unbounded, which does not mesh well with Sidetree. Having the metadata as fields in the graph data would add considerable overhead to the operation of the protocol, several orders of magnitude in the case of large metadata collections. The fact that these metadata files are stored on IPFS means that querying content by metadata requires an indexing layer to be practical. Such an implementation is comparatively simple when the data is openly available and content addressable. For a given version of a research object or component, there is a definitive state bound to that version, meaning it can never go stale. For free-floating community curated metadata payloads, these can be indexed separately to allow for choosing whether they should be included in the search or not. A rough sketch of the implementation of an index: * The service listens for protocol events updating metadata * For each update where the metadata CID has changed, fetch the new content from IPFS * Update the index for the given target with the new metadata entries With this dataset constructed, arbitrary queries can be run over the metadata. Searching metadata by free text or by schema would both be possible. # Deterministic resolution Deterministically resolving an entity instance, including particular versions To enable persistent resolution of nodes by address, we need to define an algorithm for deterministically traversing the graph. This definition is described assuming little more than functional Sidetree nodes able to find state at given commits, and the implementation of a resolver will differ depending on the practical applications used. For resolution, we use both the node `ID` and the version `Commit`. Both are unique persistent identifiers by Sidetree definition, and uniquely resolvable due to how Sidetree deterministically resolves conflicts and cryptographically ensures document update ordering. {% hint style="info" %} Some of the resolution steps laid out below linear in size complexity, while they may well be more or less constant time given a properly maintained index. {% endhint %} {% hint style="info" %} This section talks about *protocol-native* addressing, which something like [dPID](https://www.dpid.org/) can use for resolution. This is not necessarily the same thing, depending on how dPID chooses to structure addressing. {% endhint %} ## Resolution cases These cases define one step of addressing, grouped by type of target. However, a link is ideally done directly to a particular version of a node. This means using the unique ID and commit of an attestation to resolve that node, instead of addressing it relative to its target. This way of addressing allows for, more or less, instant lookups where relative addressing can contain multiple resolution steps to resolve. ### Root node Addressing the latest state of some node `N`: 1. Query network for status of node `N` ### Particular version Addressing a particular commit `C` of a node `N`: 1. Query network for status of node `N` at commit `C` ### Particular time Addressing the state of a node `N` as of time `T`: 1. Query network for update history of node `N` 2. Find the newest commit `C` that was anchored before or at time `T` 3. Resolve node `N` at commit `C` ### Particular version index Addressing a particular version `k` of a node `N`: 1. Query network for update history of node `N` 2. Select commit `C` at index `k` in update history 3. Resolve node `N` at commit `C` ### Outgoing edge Addressing of an outgoing edge from a node `N` made against some other node: 1. Resolve `N` 2. Get value `R` from reference field 3. Resolve node `R` ### Versioned outgoing edge Addressing a versioned outgoing edge from a node `N` made against some other node: 1. Query network for status of node `N` 2. Get value of reference field `R` and version field `C` 3. Resolve node `R` at commit `C` ### Incoming edges Addressing of an incoming edge to node `N`, from some node `N2` of entity type `T`: 1. Query network for all nodes of type `T` 2. Find node `N2` with 1. Reference field set to `N1` 3. Resolve `N2` {% hint style="info" %} An example of this is components or annotations pointing to a research object, and we address it from the perspective of the research object. {% endhint %} ### Versioned incoming edges Addressing of an incoming edge to node `N` as of version `C`, from some node `N2` of entity type `T`: 1. Query network for all nodes of type `T` 2. Query network for update history `U` of node `N` 3. Find versions `V` of node `N2` with 1. Reference field set to `N` 2. Version field value in `U` before version `C` 4. Resolve `N2`, considering `V` the update history while targeting `N` at `C` {% hint style="info" %} An example of this is addressing all attestations made up until a certain point for a particular research object. {% endhint %} ### Versioned data DAG paths Addressing a DAG node through UnixFS path `P`, in research object `N` as of version `C`: 1. Resolve `N` at version `C` 2. Get value `DAG` in manifest CID field 3. While `P` not empty 1. Pop first segment `S` from `P` 2. Set `DAG' = lookup(S, DAG)` 3. Loop with `DAG'` as `DAG` 4. `DAG` is now the addressed node or leaf {% hint style="info" %} Data DAG path addresses should always specify a particular research object version. {% endhint %} # Ceramic & ComposeDB The choice of underlying data and indexing layer {% hint style="info" %} Codex is currently transitioning from the now outdated `js-ceramic` & `composedb` to `rust-ceramic`. This information will be updated when that work is finished. {% endhint %} ## Ceramic Network The Ceramic Network has been chosen as the standard implementation of the data protocol, as it fulfills the necessary requirements from Sidetree, but for JSON data, and includes schema validation as part of the node message processing. Ceramic is a decentralized protocol for creating and updating mutable content on the web. This is a short summary of how Ceramic fulfills the needs for representation of the protocol entities: 1. **Streams**: At the heart of Ceramic are streams – sequences of operations on JSON documents. Each stream has a unique identifier (`StreamID`) and a cryptographically verifiable version history, allowing for updates and revisions over time. We specifically use the `StreamTile` document. 2. **Decentralized Identifiers (DIDs)**: Ceramic uses DIDs for user and data authentication. This means that any update to a stream is cryptographically linked to a DID, ensuring the integrity and provenance of data. 3. **Commit Log**: Every update to a stream in Ceramic is saved as a commit in its commit log. This allows applications and users to query the state of the stream at any point in its history, ensuring full auditability, because every version is uniquely identifiable by its `CommitID`. 4. **Pinning and Storage**: Ceramic uses the IPFS for storing the commit logs. This decentralized storage ensures data permanence and availability. 5. **Schema Enforcement**: Ceramic allows you to enforce schemas on your streams. This can be useful if you want your JSON data to adhere to specific structures or formats, which is what we need to instantiate the protocol entities. 6. **Interoperability**: Ceramic is built on standard protocols like DIDs and IPLD, which ensures a high degree of interoperability with other Web3 systems and data structures. 7. **Decentralization and Security**: Ceramic operates atop various blockchain systems and uses cryptographic methods to ensure data integrity and security. This makes it resistant to censorship and tampering. This means we can use several potential anchoring layers. ## ComposeDB What Ceramic on it own *doesn't* provide is a way to index, discover, organize, and query for existing streams (nodes) depending on which schema (entity) they implement, or track references made to other streams. ComposeDB is a type of graph database built on top of Ceramic to achieve all of these properties.\ \ In ComposeDB, we talk about models and nodes instead of schemas and streams. Relations between nodes are called edges, completing the graph analogy. 1. **Model indexing:** ComposeDB will discover and index all streams implementing a particular schema, allowing us to easily query for data without knowing about the streams beforehand. 2. **Graph model:** The data is naturally structured as a graph, where model instances (nodes) can hold edges to other entities. This is how we can query for the content of the `Reference` field type target. 3. **GraphQL API:** Automatically generated GraphQL definitions and a built-in GraphQL server allows easy data access. 4. **Composability:** the way relations are modeled allows retroactively adding indexes to old models without having to re-define them, which is very powerful as it allows the protocol implementation to be easily extended with new entities. ## Mapping The entity schemas listed generic types for references and version identifiers, as well as other terms. This is how these map to Ceramic and ComposeDB. | Entity field type | ComposeDB type | | ----------------- | ------------------------------------------------- | | `ID` | `StreamID` | | `Commit` | `CommitID` | | Network node | ComposeDB or Ceramic node, depending on the query | ## Motivation To our knowledge, there exists no other Sidetree-esque protocol which allows arbitrary schema validation, automatic indexing and data discovery, append-only operation on mutable references with full historical provenance. It also implements relations in a way that natively supports graph models both from the construction and query perspective. In addition to this, an automatically generated, self-documenting GraphQL API. Even if organizations with capable engineering departments could build the corresponding feature on top of some other data layer, having batteries included absolutely obliterates the bar of participation in building applications and services on top of the protocol. Another requirement has been removing the need for 1-1 mapping between user actions and blockchain calls, for scalability, cost, and UX reasons. Another central aspect of ComposeDB, which is hinted at in the name, is composability. There is native support for re-using models defined by others in other contexts, which means anyone can extend the protocol by creating a new type of model which can seamlessly coexist and interface with the existing graph. If such an extension gets enough community support, it can be officially added to the protocol with ease. # Implementation The implementation includes data models representing the graph entities and relations, a test suite for ensuring functionality, the Codex node services, and related software. The code is available in the [DeSci Codex repository](https://github.com/desci-labs/desci-codex), and is made available under the MIT license. # Current limitations Limitations in capabilities pending solutions As this is an early version of the protocol, some simplifying assumptions have been made. Also, some kinks and missing capabilities have not been ironed out yet. There seems to be a way forward for all of these issues, however. The sub-pages in this section digs into these problem. # Invalid data Voluntarily inflicted self-harm While it is possible for an actor to create semantically incorrect entries in the graph, this is not taken into consideration when discussing the mechanics of querying the graph. For example, there is no way of preventing non-resolvable data DAG from the properties of IPFS and IPLD. However, if it was correct at one point, that particular version can still be resolved retroactively. That means that any reference made to a semantically correct state can never break due to this limitation. It is the responsibility of gateway operators to make sure the data that is created complies with the semantics of the protocol. This means correctly computing DAG paths, updating components as the user makes changes to the DAG, et cetera. It's still absolutely possible for an actor to interact directly with the protocol and create unresolvable DAG's or nonsense nodes, but a gateway is also free to build their own definition of what is reliable data. Perhaps that just consists of nodes created from DID's with an associated ORCiD handle, profiles with an attestation from community moderators, or similar rules. The protocol doesn't have an opinion. # Timestamps When the sand of time is gravel sized Contributions to the data graph do not have an exact time associated, as they are only periodically anchored on a blockchain. This is the only notion of wall-clock time that is available in Sidetree protocols. The time between Ceramic anchor events is still measured in tens of hours, which makes it a bit hard to decide between order of events when rendering a gateway. This is a temporary problem, as the anchoring interval is intended to increase in frequency. The naive solution of adding a date time field to models isn't sufficient, as this would only be possible to validate as somewhere between the previous and next anchor event regardless. The Ceramic Anchor Service, or CAS, is a centralized system at the time of writing. There is a clear intention to decentralize this over time, which is an important change because it enables gateway operators to anchor private commits to provide verifiable history for unpublished content. Some valuable applications of this are annotation-driven anonymous review, historically verifiable draft-mode in publishing gateways, and similar features. # Delegation of permissions Allow helping hands Some planned features depend on the ability to delegate control over certain operations, like allowing someone to publish data on your behalf. For instance, letting members of an organization issue attestations on behalf of the organization DID, inviting a data steward to update the data DAG for a publication, et cetera. This is a hard problem in the decentralization space as identity circulates around the cryptographic wallet and solutions like multisigs and ring signatures are hard to work with. But there are two main ways forward. ## Gateway operation batching Gateways could implement advanced features to allow organizations to manage members, and through this collect suggested operations from individuals, which are compiled into a set that can be accepted and published by someone with control over the organization DID. This pattern of collecting suggestions that are ultimately applied by the controlling DID could be used to solve many problems regarding multi-author collaboration. The gateway can help organize the changes, illustrate the effects of application for the decision maker, and aid in adding relevant contributor relations if necessary. ## Account abstractions Instead of using a traditional single-user wallet to approve changes to a node, the DID could be controlled by what's called an account abstraction. This is a wallet controlled by a smart contract, which allows more or less arbitrary logic to regulate which keys can perform different types of actions. This could be a way to implement multi-author publishing, delegate control over publishing, and allowing a user DID to “impersonate” an organization. # Roadmap Roadmap for protocol and related services ## 2025 Q4 * Support for listing directory content of publications directly in the dPID resolver * Migration of [DeSci Nodes](https://nodes.desci.com), [dPID resolver](https://dpid.org/browse) and associated software from js-ceramic and ComposeDB to [rust-ceramic](https://github.com/ceramicnetwork/rust-ceramic) * This greatly improves stability and resilience of the network, speeds up indexing and queries, and enables network participation with Codex Nodes. * Public dashboard for Codex network health and data replication metrics ## 2026 Q1 * Public leaderboard for Codex node operators to incentivise community contribution # Feature wishlist Wanted functionality that is still further away ## Fork & merge Being able to fork a publication, make changes, and send it to the original author as a merge request would enable git-like collaboration. By having a relation created by the forker to indicate provenance and a relation created by the merger to indicate application, there would be a branching history to explore behind collaborative research projects. Possibly a prerequisite to this, depending on implementation, would be [CRDT](https://crdt.tech/) representation of content to make sure it can be cleanly applied. ## Nanopublications The protocol can represent nanopublications in different ways already: 1. A slim research object containing small files for each publication 2. Tied to larger research objects as annotation metadata payloads However, the usefulness of nanopublications at scale depends on being able to expressively query and calculate probability distributions over the assertions. There may be a need for purposeful entities built for this purpose, being able to represent both contextual (i.e., breakdowns) and stand-alone assertions in the same data type. This would make it easier to query nanopubs specifically, without needing to consider the rest of the graph at that stage. ## Organization management Being able to manage organizations, its members, and delegation of power would enable very interesting applications. Parts of this can be done in the gateway layer, but there may need to be support for certain representations in the protocol as well. ## Multi-author publication control Ways of having multiple DID's co-author nodes, and inviting someone to edit your nodes with you. This could be considered more bound to the DID provider than the protocol, but Sidetree implementations are sensitive to [late publishing attacks](https://identity.foundation/sidetree/spec/#late-publishing). Normally this doesn't pose much of a threat since only the document owner can attack itself, but if multiple actors control the document, they need to have a large degree of trust between them. Potentially, this is resolvable by an account abstraction contract which emits events for the operations, which can be picked up by network nodes and applied automatically and hence limiting the window of attack. ## Retroactive public goods funding Standardized way to reward contributions to the scientific record, creating a new way to incentivize people to “do the right thing”. ## Research object service economy Using the fact that our DIDs rely on cryptographic wallets, the protocol should enable a thriving marketplace for research object services. These could be based around writing, validation, reproducibility, data conversion, or even physical services like lab analysis.