描述
<h2>Abstract</h2><p>Blockchain has completely transformed the way we view trust and decentralized systems. However, decentralized applications face challenges when it comes to securely and reliably organizing as well as automating data in an off-chain context. This is where programmable zkOracle comes in - a new type of decentralized oracle network that utilizes zero-knowledge proofs to bring on-chain data to off-chain context, perform off-chain computation for on-chain context, and verify off-chain proofs in any environment.</p><p>In this article, we introduce Hyper Oracle, a programmable zkOracle network that aims to address the limitations of existing middle layer solutions and enable a new wave of decentralized applications.</p><h2>0. Definition of zkOracle</h2><h3>a) Framing of Oracles</h3><p>When people hear the term "oracle," they often associate it with the price feed oracle, which provides off-chain data to on-chain smart contracts. However, this is just one type of oracle among many.</p><p>A straightforward explanation of the Oracle concept, as outlined in this <a href="https://chain.link/education/blockchain-oracles" rel="noopener noreferrer" target="_blank">educational resource</a>, divides it into two main types:</p><ul><li>Input Oracle: delivers off-chain data to on-chain context (ex: Chainlink Price Feeds).</li><li>Output Oracle: delivers on-chain data to off-chain context for advanced computation (ex: Hyper Oracle zkIndexing).</li></ul><p class="ql-align-center"><span style="color: transparent;"><img src="https://mirror.xyz/_next/image?url=https%3A%2F%2Fimages.mirror-media.xyz%2Fpublication-images%2FYlglONJVADsUQThhBJo_u.png&w=3840&q=75" alt="Input Oracle and Output Oracle" height="1440" width="2560"></span></p><p>Input Oracle and Output Oracle</p><p>In the realm of blockchain, the terminology "input" and "output" are used to distinguish between two types of oracles: input oracles and output oracles. In addition, Hyper Oracle is defining the I/O oracle, a specialized type of oracle that integrates both input and output oracles by first following the output oracle's flow and then the input oracle's. Each oracle can be further broken down into three components: data source, computation, and output.</p><ul><li>Input Oracle</li><li class="ql-indent-1">Data Source: Off-chain data (e.g. CEX price feeds, real-world weather data)</li><li class="ql-indent-1">Computation: Aggregation of off-chain data and "uploading" of data</li><li class="ql-indent-1">Output: On-chain data (equivalent to off-chain data, but stored on-chain)</li><li>Output Oracle</li><li class="ql-indent-1">Data Source: On-chain data (e.g. smart contract interactions or events like ERC-20 transfers or ERC-721 minting)</li><li class="ql-indent-1">Computation: Indexing, aggregation, filtering, or other complex computation</li><li class="ql-indent-1">Output: Off-chain data in an organized and easy-to-use form</li><li>I/O Oracle</li><li class="ql-indent-1">Combines Input Oracle and Output Oracle with Output flow first, then Input flow.</li></ul><p>In summary, oracles are software programs that handle information within the context of a blockchain. Further information about Hyper Oracle zkAutomation and zkIndexing will be discussed in subsequent sections.</p><p class="ql-align-center"><span style="color: transparent;"><img src="https://mirror.xyz/_next/image?url=https%3A%2F%2Fimages.mirror-media.xyz%2Fpublication-images%2FnCmJHe3yX6Zwq7nQQixdg.png&w=3840&q=75" alt="Comparison of Oracle Types" height="1440" width="2560"></span></p><p>Comparison of Oracle Types</p><h3>b) ZK (Zero-Knowledge)</h3><blockquote>'<a href="https://www.bcbgroup.com/what-does-trustless-mean-in-crypto/#:~:text=%27Trustless%27%20refers%20to%20a%20quality,trust%20in%20a%20third%20party.%20." rel="noopener noreferrer" target="_blank">Trustless</a>' refers to a quality of a decentralised blockchain, whereby in using the network there is no need to rely on trust in a third party.</blockquote><p>Zero-knowledge protocols or proofs ensure the validity of computation as explained in this <a href="https://medium.com/starkware/how-can-cryptographic-proofs-provide-a-guarantee-of-financial-solvency-79ddc333116f" rel="noopener noreferrer" target="_blank">article</a>. They allow us to prove that the computation is correct without relying on external trust. As verifiers, we can be sure that the output data is also correct.</p><p>With zk, the computation outputs both the final computational result and a zk proof. You can think of zk proofs as a receipt that is always correct. We can verify the short receipt to ensure that all the data (items bought) and computation (summation of item costs) are correct.</p><p>The benefits of zk proofs are:</p><ul><li>Easy to verify (no need for complex consensus to agree on data or to re-run computation with full data; verification can be performed in any context, leading to interoperability and composability possibilities)</li><li>Faster finality (once data and zk proof are verified, they can be considered fully correct and finalized)</li><li>Highest security (purely based on cryptography and math)</li><li><a href="https://mirror.xyz/msfew.eth/gBVAvKt_G6y0ns5eA5vjZYEBgWdvPRu6KI9CujUvsek" rel="noopener noreferrer" target="_blank">Simpler system and mechanism</a> (no need to learn about complicated tokenomics or mechanisms)</li><li><a href="https://dl.acm.org/doi/pdf/10.5555/88314.88333" rel="noopener noreferrer" target="_blank">Can be used for any computation</a></li></ul><p>In summary, zk is a powerful cryptographic stack for trustless and secure computation.</p><h3>c) Issue of Traditional Oracle Network</h3><p>Traditional oracle networks, such as Chainlink or The Graph, have typically created new staking networks, rather than extending the base layer. These networks rely on tokenomics and staking mechanisms to ensure overall security and decentralization. However, this approach results in several issues:</p><ul><li>Security</li><li>Lack of Uniformity with Base Layer Blockchain:</li><li class="ql-indent-1">These networks create separate new networks in addition to the base layer blockchain. They cannot inherit the established security level and mature ecosystem model of Ethereum. In practice, they are much less secure than base layer blockchains such as Ethereum.</li><li>Uncertain Trust Established by Tokenomics:</li><li class="ql-indent-1">The security of these networks depends on variables such as the number of nodes and the number of honest operators in the network. It is difficult to determine the actual level of trust and security of the oracle network. Additionally, the security of the network is tied to tokenomics, introducing another risk factor that comes with usually high trading activity of a network's token.</li><li>Low Cost and High Profit of Attack:</li><li class="ql-indent-1">These networks often serve as the primary component of decentralized applications with high accrued value. The potential profit of attacking these protocols by attacking the oracle network is high, while the cost of attacking the oracle network is low if its token price or the staking requirement is too low.</li><li>Decentralization</li><li>High Network Participation Barrier:</li><li class="ql-indent-1">The more nodes or parties participate in a network, the better the decentralization level. However, in some traditional oracle networks, the barrier or lowest staking amount is too high for normal actors to afford participation. For example, The Graph requires about <a href="https://thegraph.com/docs/en/network/indexing/#what-is-the-minimum-stake-required-to-be-an-indexer-on-the-network" rel="noopener noreferrer" target="_blank">100K GRT (US$16000 as of 3/1) to simply start an indexer node</a> without considering hardware requirements, and <a href="https://thegraph.com/docs/en/network/indexing/#what-are-disputes-and-where-can-i-view-them" rel="noopener noreferrer" target="_blank">10K GRT (US$1600 as of 3/1) to challenge and report a wrong behavior</a>. These high barriers make it difficult for spontaneous and honest nodes to participate in network activities. Additionally, starting up a new network requires large capital for initial staking.</li><li>Token Distribution Centralization:</li><li class="ql-indent-1">Usually, the entities behind these networks control a large portion of token supply. A metric to measure the decentralization level of a network is the Nakamoto Coefficient. It represents the smallest number of independent entities that can act collectively to shut down a blockchain. A larger coefficient may indicate a more decentralized network. However, <a href="https://nakaflow.io/" rel="noopener noreferrer" target="_blank">only 4 entities are needed to shut down The Graph's network</a>. The centralization of token distribution makes the tokenomics and overall mechanism centralized.</li><li>Efficiency</li><li>Overhead in Performance:</li><li class="ql-indent-1">The most critical factor of an oracle network is performance. A traditional distributed network usually increases latency as the number of nodes in the network grows. If it’s not a 1-of-N trust model (”<a href="https://vitalik.ca/general/2020/08/20/trust.html" rel="noopener noreferrer" target="_blank">the system works as long as at least one of them does what you expect them to.</a>”) like zk network, it’s hard to both achieve decentralization (more nodes), and performance (relatively less nodes).</li><li>Waste of Computation Power:</li><li class="ql-indent-1">Traditional decentralized networks with PoW mechanism have a lot of redundant computation and wasted computing power. These are used in the consensus generation process to protect the overall security of the network. However, this wasted computing power also represents a higher cost of operation and service usage.</li></ul><h3>d) zkOracle</h3><p>A zkOracle addresses and resolves the aforementioned issues with the following solutions, including:</p><ul><li>Providing a unstoppable autonomous network</li><li>Math as the consensus</li><li>Safeguarding the security of the base layer</li><li>A 1-of-N trust model</li><li>Optimal cryptography-native decentralization</li><li>Efficient computing power allocation (ideally no excess wasted)</li></ul><p>In later sections, we will compare traditional oracle networks with zkOracle network, detailing their differences.</p><p>As a component that processes data, an oracle must ensure both the accuracy and security of computation. It is important to confirm that the output is valid and correct and that the verification process is fast (sublinear).</p><p class="ql-align-center"><span style="color: transparent;"><img src="https://mirror.xyz/_next/image?url=https%3A%2F%2Fimages.mirror-media.xyz%2Fpublication-images%2Fs_LQyIjFiUGqArs_swdKr.png&w=3840&q=75" alt="Traditional Oracle" height="1440" width="2560"></span></p><p>Traditional Oracle</p><p>To achieve a trustless and secure oracle, we need to make it a zkOracle.</p><p>Hyper Oracle zkOracle is natively categorized as output zkOracle and I/O zkOracle.</p><p><strong>I. Output zkOracle</strong></p><p>An output zkOracle is an output oracle that uses zk to prove its computation’s validity. An example of this is Hyper Oracle zkIndexing Meta App.</p><p class="ql-align-center"><span style="color: transparent;"><img src="https://mirror.xyz/_next/image?url=https%3A%2F%2Fimages.mirror-media.xyz%2Fpublication-images%2FxGXYBvDKXLCspH-kTTgne.png&w=3840&q=75" alt="Output zkOracle" height="1440" width="2560"></span></p><p>Output zkOracle</p><ul><li>Data Source: On-chain Data</li><li>The straightforward solution is to use on-chain data as the source. This data has already been verified and secured by the blockchain. Off-chain data sources cannot efficiently reach the trust level of on-chain data (at least not yet, according to <a href="https://andrecronje.medium.com/oracle-evolution-ab7ce23da15b" rel="noopener noreferrer" target="_blank">this source</a>). The on-chain data source solution requires zkOracle to act as an output oracle.</li><li>Computation: Execution and ZK Proof Generation</li><li>The solution is to create a zk proof of the computation (typically indexing, aggregation, and filtering…) and enable the step of accessing the data source in a zero knowledge fashion. This adds a layer of validity and trustlessness to the computation. The output will now be accompanied by a zk proof, making the computation and output verifiable.</li><li>Output: Execution Output and On-chain Verifiable zk Proof</li><li>The output of the computation will be both the execution output and a verifiable zk proof. The proof can be easily verified in a smart contract or any other environment. The verification component can confirm the validity of the execution of the zkOracle.</li></ul><p><strong>II. I/O zkOracle (Output + Input)</strong></p><p>An I/O zkOracle is an output oracle and an input oracle both with ZK as computation. An example is Hyper Oracle zkAutomation Meta App.</p><p class="ql-align-center"><span style="color: transparent;"><img src="https://mirror.xyz/_next/image?url=https%3A%2F%2Fimages.mirror-media.xyz%2Fpublication-images%2FLgrz3U24nryB-wgc5am9b.png&w=3840&q=75" alt="I/O zkOracle" height="1440" width="2560"></span></p><p>I/O zkOracle</p><p>In this case, zkOracle will function as a combination of two oracles that operates in two stages:</p><ul><li>Data Source: On-chain Data</li><li>The data source for I/O zkOracle is identical to the output zkOracle.</li><li>Computation: Execution and ZK Proof Generation</li><li>The computation of I/O zkOracle includes the output zkOracle (which involves indexing, aggregation, and filtering) as well as the input zkOracle (which involves setting up off-chain computation results as calldata for smart contract calls). The combination of both parts makes it feasible to automate smart contracts with complex off-chain computation.</li><li>Output: On-chain data and On-chain Verifiable zk Proof</li><li>The output for this stage includes on-chain data which is the execution output provided on-chain as calldata, and a verifiable zk proof. This proof is easily verifiable in smart contracts or any other environment. The verification component can confirm the validity of the execution of I/O zkOracle.</li></ul><p><strong>III. Definitions</strong></p><p>Technically, zkOracle is an oracle with verifiable pre-commit computation.</p><p>Functionally, zkOracle utilizes zk to ensure the computation integrity of the oracle node for the oracle network's security, instead of staking and slashing mechanism.</p><p>In essence, zkOracle is an oracle that utilizes zk for computation and data access, while also using on-chain data for the data source to secure the oracle in a trustless manner.</p><h2>1. zkOracle Network</h2><blockquote>zkOracle = zkPoS + zkGraph run in zkWASM</blockquote><h3>a) zkOracle Overview</h3><p>Hyper Oracle is a network of zkOracles designed for blockchains. At present, the zkOracle network operates solely for the Ethereum blockchain. It retrieves the data from every block of the blockchain as a data source with zkPoS and processes the data using programmable zkGraphs that run on zkWASM, all in a trustless and secure manner.</p><p>Here is the zkOracle design for the Ethereum blockchain. This serves as a foundational design for a zkOracle, complete with all of the essential components.</p><p class="ql-align-center"><span style="color: transparent;"><img src="https://mirror.xyz/_next/image?url=https%3A%2F%2Fimages.mirror-media.xyz%2Fpublication-images%2FwTqbjPNBgDeaRz-DiTtB2.png&w=3840&q=75" alt="zkOracle for Ethereum" height="1440" width="2560"></span></p><p>zkOracle for Ethereum</p><p>zkPoS verifies Ethereum consensus with a single zk proof that can be accessed from anywhere. This allows zkOracle to obtain a valid block header as a data source for further processing.</p><p>zkWASM (zkVM in the graph) is the runtime of zkGraph, providing the power of zk to any zkGraph in the Hyper Oracle Network. It is similar to the kind of zkEVM used in ZK Rollups.</p><p>zkGraph (run in zkWASM) defines customizable and programmable off-chain computation of zkOracle node’s behaviors and Meta Apps. It can be thought of as the smart contract of the Hyper Oracle Network.</p><h3>b) zkPoS</h3><blockquote>Trustless Block Header Fetching, the “Data Source” in Hyper Oracle Network</blockquote><p>One crucial step in implementing a zkOracle is to retrieve blockchain data for the data source, specifically block header data. Block header data serves as the essential entry point for obtaining the actual data (the three roots) needed for the zkOracle.</p><p class="ql-align-center"><span style="color: transparent;"><img src="https://mirror.xyz/_next/image?url=https%3A%2F%2Fimages.mirror-media.xyz%2Fpublication-images%2FaSQotZTtxj69UyCOY7JAZ.png&w=3840&q=75" alt="Ethereum Block Header Structure" height="1440" width="2560"></span></p><p>Ethereum Block Header Structure</p><p>There are different methods to acquire a block header, but the simplest and least secure or decentralized way is to obtain it from a trusted source, such as Infura. Another option is to use current light clients like <a href="https://a16zcrypto.com/building-helios-ethereum-light-client/" rel="noopener noreferrer" target="_blank">Helios</a>.</p><p>The solution for trustless block header fetching is zkPoS, which proves Ethereum's consensus with zk. Combining zkPoS and light clients like Helios, we can build a SNARK-based light client that uses off-chain computation to eliminate most of the computation of client verification..</p><p>To maintain its trustless nature, zkPoS for a zkOracle presents the following challenges:</p><ul><li>Constant (short) verification time</li><li>Constant (small) proof size</li><li>Performant proof generation</li><li>No external trust to third-party</li></ul><p>Hyper Oracle zkPoS will deliver a zk light client and address the above challenges by:</p><ul><li>SNARKifying block attestation and other logics of Ethereum consensus (see graph below)</li><li>Recursive proof for multiple blocks of Ethereum consensus</li></ul><p class="ql-align-center"><span style="color: transparent;"><img src="https://mirror.xyz/_next/image?url=https%3A%2F%2Fimages.mirror-media.xyz%2Fpublication-images%2FmWjXbBbuXS6khpe847If7.png&w=3840&q=75" alt="zkPoS" height="1440" width="2560"></span></p><p>zkPoS</p><p>For more details, please refer to our previous blog post: <a href="https://mirror.xyz/hyperoracleblog.eth/lAE9erAz5eIlQZ346PG6tfh7Q6xy59bmA_kFNr-l6dE" rel="noopener noreferrer" target="_blank">zkPoS: End-to-end Trustless</a>.</p><p>zkPoS will enhance the end-to-end trust minimization of Hyper Oracle zkOracle by providing a trustless data source as the input for the oracle.</p><h3>c) zkGraph</h3><blockquote>Customizable Off-chain Computation, the “Smart Contract” in Hyper Oracle Network</blockquote><p><strong>I. Intro to zkGraph</strong></p><p>zkGraph defines the off-chain computation of Hyper Oracle nodes, including data-related behaviors and zk proof generation, much like how smart contracts define the EVM computation of Ethereum nodes.</p><p class="ql-align-center"><span style="color: transparent;"><img src="https://mirror.xyz/_next/image?url=https%3A%2F%2Fimages.mirror-media.xyz%2Fpublication-images%2FPhwmvy-7qPqTorMN6mK7J.png&w=3840&q=75" alt="Hyper Oracle zkGraph and Ethereum Smart Contract" height="1440" width="2558"></span></p><p>Hyper Oracle zkGraph and Ethereum Smart Contract</p><p>Smart contract developers can build both the smart contract and the zkGraph. Users can then interact with both.</p><p>To utilize the infrastructure of Hyper Oracle Meta Apps, developers must configure and code their zkGraph to specify how they want the data to be handled. The Hyper Oracle nodes then process the data and generate a zk proof based on the specified definitions.</p><p>In summary, a zkGraph is a program that defines the mapping of data (after the data preprocessing of zkPoS) and configures Meta Apps.</p><p><strong>II. Develop zkGraph</strong></p><p>A zkGraph is customizable and programmable, consisting of three main components that can be fully customized:</p><ul><li>Manifest (zkgraph.yaml): The data source, used to configure information such as the Meta Apps used, the target blockchain network, and the target smart contract.</li><li>Schema (schema.graphql): The data structure, used to define how data is stored and accessed.</li><li>Mapping (mapping.ts): The data mapping (Off-chain Computation), used to compute blockchain data into other forms.</li></ul><p class="ql-align-center"><span style="color: transparent;"><img src="https://mirror.xyz/_next/image?url=https%3A%2F%2Fimages.mirror-media.xyz%2Fpublication-images%2F1qtPVVUV_JeDvPXOa6Fed.png&w=3840&q=75" alt="zkGraph Components" height="1440" width="2560"></span></p><p>zkGraph Components</p><p>The core of a zkGraph is the mapping (mapping.ts) file. The code defines the off-chain computation program.</p><p>The mapping file usually defines handlers for filtering on-chain events or setting up calldata of smart contract automation. The filters are run in zkWASM (details in the next section), and the zk proofs are generated to ensure computational integrity and validity.</p><p>For deployment, all code files for zkGraph will be stored in EthStorage, which is a storage scaling layer supported by Ethereum ESP. This will guarantee that development pipeline for zkGraph is fully decentralized.</p><p>Below is the sample code of a zkGraph (for illustrative purposes only):</p><p class="ql-align-center"><span style="color: transparent;"><img src="https://mirror.xyz/_next/image?url=https%3A%2F%2Fimages.mirror-media.xyz%2Fpublication-images%2FgveEfaBfzp2VZMnyY1uJ7.png&w=3840&q=75" alt="zkGraph Sample Code" height="800" width="2560"></span></p><p>zkGraph Sample Code</p><p><strong>III. zkGraph’s Subgraph-Equivalence</strong></p><p>It is a common practice to avoid reinventing the wheel and instead <a href="https://twitter.com/0xzuberg/status/1613912873486651393" rel="noopener noreferrer" target="_blank">leverage the existing ecosystem</a>. zkGraph is the equivalent to The Graph's Subgraph.</p><p>Migrating existing Subgraphs (over 600 and growing) to zkGraph requires just 10 lines of configuration difference. Implementations such as <a href="https://github.com/messari/subgraphs" rel="noopener noreferrer" target="_blank">Standardized Subgraph</a> and ecosystem tooling like <a href="https://docs.goldsky.com/indexing/instant-subgraphs" rel="noopener noreferrer" target="_blank">Instant Subgraph</a> and <a href="https://thegraph.academy/developers/subgraph-uncrashable/" rel="noopener noreferrer" target="_blank">Subgraph Uncrashable</a> can be used for developing zkGraph.</p><p>With zkWASM, zkGraph supports the general computation of any AssemblyScript (~TypeScript) syntax. In other words, Hyper Oracle Meta Apps are fully customizable and programmable.</p><h3>d) zkWASM</h3><blockquote>Power zkGraph with ZK</blockquote><p>Since zkGraph is fully customizable and programmable, it requires a general runtime environment to execute. Similar to how EVM runs smart contracts, zkWASM (zkVM of WebAssembly) runs zkGraphs.</p><p>As a part of the Subgraph-Equivalence, zkGraph also employs WASM-based mappings. It is "translated" into AssemblyScript, a language designed to be run in a WebAssembly runtime environment.</p><p>The Graph uses wasmtime as the WebAssembly runtime for Subgraph, while Hyper Oracle uses zkWASM as the WebAssembly runtime for zkGraph. This difference in execution engines makes zkGraph zk, without sacrificing any general computation ability. Replacing wasmtime with zkWASM is like replacing EVM with zkEVM. Any program run in zkWASM has the superpower of ZK, including verifiability, trustlessness, decentralization, and computational integrity.</p><p class="ql-align-center"><span style="color: transparent;"><img src="https://mirror.xyz/_next/image?url=https%3A%2F%2Fimages.mirror-media.xyz%2Fpublication-images%2FzMMEeHOgxy38pdzhzVMFT.png&w=3840&q=75" alt="Hyper Oracle zkWASM and The Graph WASM" height="1440" width="2560"></span></p><p>Hyper Oracle zkWASM and The Graph WASM</p><p>Hyper Oracle zkGraph are not limited to be running exclusively in zkWASM. We value the decentralization of prover/client diversity and are constantly exploring new ways to power zkGraph with zk.</p><p>To learn more about zkWASM, please take a look at our previous blog post: <a href="https://mirror.xyz/hyperoracleblog.eth/abKqUB4iEJ4kRsGqq8baIFUnhV_eY-lblmhCrwRm31E" rel="noopener noreferrer" target="_blank">zkWASM, The Next Chapter of ZK and zkVM</a>. For further technical details, you can refer to the paper titled <a href="https://jhc.sjtu.edu.cn/~hongfeifu/manuscriptb.pdf" rel="noopener noreferrer" target="_blank">ZAWA: A ZKSNARK WASM Emulator</a>.</p><p>zkWASM will enable Hyper Oracle zkGraph to achieve programmable configuration, general computation, subgraph equivalence, and the superpower of zero-knowledge proofs.</p><h3>e) zkOracle in Hyper Oracle Network</h3><p>Here is the conceptual architecture for a running zkOracle node in the Hyper Oracle network.</p><p>The design at the start of this section is similar to the previous one, with a few changes made to highlight certain details.</p><p class="ql-align-center"><span style="color: transparent;"><img src="https://mirror.xyz/_next/image?url=https%3A%2F%2Fimages.mirror-media.xyz%2Fpublication-images%2FWQeSFsj9dNoyrrOzVim6v.png&w=3840&q=75" alt="zkOracle Node Structure" height="1440" width="2560"></span></p><p>zkOracle Node Structure</p><p>To discuss the design, let's move from left to right and top to bottom:</p><ol><li>Ethereum blockchain serves as the original on-chain data source for zkOracles, but in the future, any network can be used.</li><li>The Hyper Oracle zkOracle node consists of two main components: zkPoS and zkWASM.</li></ol><ul><li class="ql-indent-1">zkPoS fetches the block header and data roots of the Ethereum blockchain by proving Ethereum's consensus with zk. The zk proof generation process can be outsourced to a decentralized prover network. zkPoS works as the foreign circuit of zkWASM.</li><li class="ql-indent-1">zkPoS feeds the block header and data roots to zkWASM. zkWASM takes this data as essential inputs for running zkGraphs.</li><li class="ql-indent-1">zkWASM runs customized data mappings defined by zkGraphs and generates zk proofs of those operations. The operator of the zkOracle node can choose the number of zkGraphs they wish to run (from one to all deployed zkGraphs). The zk proof generation process can be outsourced to a decentralized prover network.</li></ul><ol><li>The output of a zkOracle is off-chain data that developers can use through Hyper Oracle Meta Apps (which are covered in the next sections). The data also comes with zk proofs that demonstrate the validity and computation of the data.</li></ol><p>Only one zkOracle node is necessary to maintain network security. In the Hyper Oracle network, there can still be multiple zkOracle nodes targeting zkPoS as well as each zkGraph. This enables parallel generation of zk proofs, which can significantly enhance performance.</p><h2>2. Meta Apps</h2><blockquote>Meta Apps = zkIndexing + zkAutomation, with zkGraph</blockquote><p>zkOracle network are the technical architecture of Hyper Oracle. Meta Apps (or zkOracle services) are the “products” of Hyper Oracle.</p><p>Hyper Oracle Meta Apps provide a range of infrastructure services that can be used by any DApp developer. As every DApp requires specific infrastructure, Meta Apps offer fully decentralized and secure options for the DApp.</p><h3>a) Fixing Infrastructure</h3><blockquote>Note: A better and more accurate term for “Infrastructure” is “Middleware”. Here we use the generic term of “Infrastructure” for ease of the understanding.</blockquote><p>A decentralized application (DApp) is more complex than just a smart contract. A complete DApp should include at a minimum:</p><ul><li>Interface (usually a website serving as the front-end)</li><li>Infrastructure/middleware component (which may include an indexing service to organize data, an automation/keeper network to automatically call functions, a price oracle to provide data, or all of these)</li><li>Smart contract</li></ul><p class="ql-align-center"><span style="color: transparent;"><img src="https://mirror.xyz/_next/image?url=https%3A%2F%2Fimages.mirror-media.xyz%2Fpublication-images%2FtCm0qRGsQ7PAxvmFrI1sM.png&w=3840&q=75" alt="DApp Architecture" height="1440" width="2560"></span></p><p>DApp Architecture</p><p>Since the introduction of smart contracts, consensus researchers and network engineers have been exploring ways to scale the network and increase the computational capabilities of smart contracts.</p><p>To achieve better performance without sacrificing the decentralization of a blockchain, the best approach currently available is to use rollups, such as Optimistic Rollup and ZK Rollups.</p><p class="ql-align-center"><span style="color: transparent;"><img src="https://mirror.xyz/_next/image?url=https%3A%2F%2Fimages.mirror-media.xyz%2Fpublication-images%2FeZz3vU_vDmjVkYWAo20Kl.png&w=3840&q=75" alt="DApp Architecture with Better Network" height="1440" width="2560"></span></p><p>DApp Architecture with Better Network</p><p>Moving forward, we must also enhance the infrastructure for DApps. It's important to note that the issues with current infrastructure protocols extend beyond scalability, encompassing concerns around decentralization, trustlessness, and security. Since infrastructure is not inherently built into the blockchain network, it cannot fully leverage the existing advantages of blockchain technology. But is there a solution?</p><p>By extending DApps' capabilities, infrastructure can become an extension of smart contract's original features, which are currently isolated on the blockchain.</p><p>To achieve the desired level of decentralization, the infrastructure must convince the blockchain network (which is not aware of it) of the infrastructure's off-chain computations.. There are two effective ways to accomplish this:</p><p>Firstly, convince L1 and then upgrade the infrastructure for decentralization:</p><ul><li>Run in smart contract. (However, infrastructure is an extension to smart contracts, so it cannot be run in smart contracts by definition.)</li><li>Provide some kind of proof. (Optimistic, ZK, <a href="https://medium.com/initc3org/complete-knowledge-eecdda172a81" rel="noopener noreferrer" target="_blank">CK</a>, or <a href="https://twitter.com/jon_charb/status/1620101172739846147" rel="noopener noreferrer" target="_blank">Modular Hybrid</a>…)</li></ul><p>We will go over the advantages of ZK over Optimistic for infrastructure in later sections. For now, we look at a more fitting solution, Hyper Oracle Meta Apps, to:</p><ol><li>Convince L1 with zk proof (<a href="https://twitter.com/backaes/status/1613856760657231872" rel="noopener noreferrer" target="_blank">a short receipt ensures computational integrity</a>).</li><li>Provide decentralized infrastructure.</li><li>Build the next generation of DApps.</li></ol><p class="ql-align-center"><span style="color: transparent;"><img src="https://mirror.xyz/_next/image?url=https%3A%2F%2Fimages.mirror-media.xyz%2Fpublication-images%2FswAEpIPtA-rjqFLSBFfF5.png&w=3840&q=75" alt="DApp Architecture with Better Infra and Better Network" height="1440" width="2560"></span></p><p>DApp Architecture with Better Infra and Better Network</p><p>In essence, Hyper Oracle Meta Apps bring innovation to infrastructure, similar to the way ZK Rollups bring innovation to blockchain.</p><h3>b) zkAutomation</h3><blockquote>Trustless Automation for Smart Contracts</blockquote><p><strong>I. Intro to zkAutomation</strong></p><p><a href="https://mirror.xyz/hyperoracleblog.eth/UYI8mpq6zJ8L2Hbqrliss0mg92v7dNAqz0UhO41d_dM" rel="noopener noreferrer" target="_blank">The blockchain world is driven by automation, bots, and keepers</a>. Automated programs are required to call smart contracts at certain intervals to maintain the AMM's optimal price flow or to keep lending protocols healthy by avoiding bad debt.</p><p><a href="https://twitter.com/0xSacha/status/1624102663557087247" rel="noopener noreferrer" target="_blank">Sometimes, these automation calls are too complex for on-chain calculation, too frequent for governance voting, and too critical for permissionless system</a>. In such cases, developers require automation protocols to provide these services, so they don't have to build and run their own automation.</p><p>However, having a universal protocol for automation and keeper functions is insufficient. The automation's execution and off-chain data source computations must be secure. If these computations are invalid, the operator nodes must be flagged and punished on the automation network.</p><p>zkAutomation is the only automation infrastructure in the market that can execute both with zero knowledge proofs. zkAutomation is not just an output zkOracle meta app, but it is also an I/O zkOracle, because the data flows from on-chain (original data) to off-chain (zkGraph source) to on-chain (automation triggered).</p><p>zkAutomation is Hyper Oracle’s trustless automation protocol based on zero knowledge proofs. zkAutomation's automation execution is entirely secured by zk, and the automation sources (data sources of the automation) and trigger conditions (when to trigger the automation) are fully customizable using zkGraphs. Developers can build bots, such as arbitrage bots, to perform on-chain trading for profit, keepers like liquidation keepers to safeguard their protocol health, or even automated protocols like on-chain ETFs or on-chain stable coins to achieve fully decentralized financial applications.</p><p class="ql-align-center"><span style="color: transparent;"><img src="https://mirror.xyz/_next/image?url=https%3A%2F%2Fimages.mirror-media.xyz%2Fpublication-images%2FBKQMF_WmZiM6tFUBLY4Pi.png&w=3840&q=75" alt="zkAutomation" height="941" width="1268"></span></p><p>zkAutomation</p><p><strong>II. Use zkAutomation</strong></p><p>Developers can use zkAutomation by deploying their automation program through Hyper Oracle's Web App. They simply need to specify their target contract, target function, and source (when to trigger). For more complex trigger conditions, developers can choose to either trigger automation every N-th block (in scenarios like a keeper bot) or use a zkGraph as the off-chain source.</p><p>zkGraphs for zkAutomation can be built from scratch, migrated from zkGraphs for zkIndexing or existing subgraphs, or by reusing deployed ones. If developers choose to migrate from an existing zkGraph for zkIndexing, they only need to change the Meta App type of that zkGraph to make it a valid source for zkAutomation.</p><p>Once the zkAutomation job has been registered (and the zkGraph has been optionally deployed), the corresponding zkAutomation service will start and automation jobs will be executed when the trigger condition is met.</p><h3>c) zkIndexing</h3><blockquote>Trustless Indexing, Querying, and Accessing of Blockchain Data</blockquote><p><strong>I. Intro to zkIndexing</strong></p><p>Accessing most on-chain data directly can be a challenge for developers. One potential solution is for developers to run their own indexer to organize the data into a more easily searchable format. However, this can be a <a href="https://twitter.com/DennisonBertram/status/1621657835334402050" rel="noopener noreferrer" target="_blank">difficult and time-consuming</a> process as it involves rebuilding the entire blockchain state and indexing events from smart contracts. This is where indexing protocols like The Graph come in handy.</p><p>As mentioned in <a href="https://a16zcrypto.com/content/article/building-helios-ethereum-light-client/" rel="noopener noreferrer" target="_blank">a blog post</a> about Helios., blockchains are used for their trustlessness, allowing self-sovereign access to wealth and data. Ethereum has delivered on this promise, but there are concessions for convenience, such as centralized RPC servers like Alchemy. These providers run high-performance nodes on cloud servers so users can easily access chain data, but users must trust them and can't verify the correctness of their queries.</p><p>Having a centralized RPC, or a universal indexing and querying protocol is not enough. It is also important to ensure that the<a href="https://twitter.com/DennisonBertram/status/1621665717274775557" rel="noopener noreferrer" target="_blank"> indexed data is accurate and reliable</a> since incorrect data can be even more problematic<a href="https://twitter.com/0xngmi/status/1567594375357546496" rel="noopener noreferrer" target="_blank"> than not having an indexer at all</a>. In other words, it is crucial to guarantee the computational integrity and security of the indexing process.</p><p>zkIndexing is a unique indexing infrastructure that excels in both accuracy and security using zero knowledge proofs. zkIndexing is a typical output zkOracle meta app, with the data flowing from on-chain (original blockchain data) to off-chain (zkGraph indexed data).</p><p>zkIndexing is Hyper Oracle’s trustless indexing protocol based on zero knowledge proofs. zkIndexing’s indexing and querying schema are fully customizable with zkGraphs. Developers can build any end-to-end decentralized application with zkIndexing.</p><p class="ql-align-center"><span style="color: transparent;"><img src="https://mirror.xyz/_next/image?url=https%3A%2F%2Fimages.mirror-media.xyz%2Fpublication-images%2FnW1z-s2Gvrt3yaq0Y6q5S.png&w=3840&q=75" alt="zkIndexing" height="708" width="1936"></span></p><p>zkIndexing</p><p><strong>II. Use zkIndexing</strong></p><p>After the deployment of a zkGraph, the corresponding zkIndexing service will start indexing the blockchain and data will be queryable for any developer.</p><p>To utilize zkIndexing, developers need to utilize a zkGraph to define the indexing behaviors and querying schema.</p><p>Developers can build zkGraphs for zkIndexing from scratch, migrate them from existing subgraphs, or reuse deployed zkGraphs. For existing protocols like Uniswap, AAVE, or Curve, there are multiple open-source implementations available.</p><p>Once the zkGraph is deployed, the corresponding zkIndexing service will start indexing the blockchain, and the data will be queryable for any developer.</p><h2>3. Comparisons</h2><h3>a) zkOracle Network vs. Other Oracle Network</h3><p>When creating a decentralized oracle, the zkOracle network has notable differences compared to traditional ones.</p><p>To start, the workflow and architecture are distinct. The zkOracle network is more streamlined since all intensive computation is carried out securely and trustlessly off-chain using ZK. In contrast, traditional oracles are restricted to trustless aggregation and "trusted" computation by a smart contract's limited computing capability.</p><p class="ql-align-center"><span style="color: transparent;"><img src="https://mirror.xyz/_next/image?url=https%3A%2F%2Fimages.mirror-media.xyz%2Fpublication-images%2Fw9RDldfwTJA5-7khkwG4y.png&w=3840&q=75" alt="zkOracle Network vs. Other Oracle Network" height="1440" width="2560"></span></p><p>zkOracle Network vs. Other Oracle Network</p><p>Based on the graph shown above, we can compare the advantages of zkOracle network to traditional oracle networks:</p><ul><li>Decentralization (Trustlessness, Security, and Censorship-Resistence)</li><li>The zkOracle network, Hyper Oracle, operates without requiring external trust in third parties, making it a trustless network. In contrast, traditional oracle networks rely on trusted third parties and networks. The zkOracle network follows the 1-of-N trust model, as defined in Vitalik Buterin's article on <a href="https://vitalik.ca/general/2020/08/20/trust.html" rel="noopener noreferrer" target="_blank">trust models</a>, which only requires one node to maintain the network's health and uptime. Traditional oracle networks, on the other hand, are only considered decentralized when they reach an extremely large number of nodes. The security of the zkOracle network is fully based on mathematics and cryptography, and it inherits its security from Ethereum, which serves as its data source.</li><li>Performance and Finality</li><li>Finality is achieved at the end of the challenge period, according to <a href="https://twitter.com/norswap/status/1613329330410504193" rel="noopener noreferrer" target="_blank">a widely-accepted definition of finality for rollups</a>. Traditional oracle and zkOracle networks can be compared to Optimistic and ZK Rollups, respectively, so we can use this definition for performance comparison. Input oracles, such as Chainlink Price Feeds, and output oracles, such as The Graph Protocol, rely on the slashing or challenge period, which can take days or even weeks. However, zkOracle networks, like Hyper Oracle, are based on the zk proof generation time. While zk proof generation has overheads compared to pure computation, the oracle's performance still improves from days or weeks of challenge period to minutes or even seconds even without considering parallel proving. Due to the nature of zk, adding more nodes to the zkOracle network can nearly <a href="https://twitter.com/toghrulmaharram/status/1629356500555628546" rel="noopener noreferrer" target="_blank">linearly boost</a> its performance with parallel proving, instead of only creating redundancy in the network.</li><li>Cost</li><li>The zkOracle network follows a 1-of-N trust model, which means that only one honest node is needed to create a secure network. This eliminates the redundancy found in traditional oracle networks, making it a more secure option. Despite the added cost of zk proving (in the case of zkEVM, proving cost for a large batch of transactions is only $0.06), the zkOracle network generates lower fees than traditional oracle networks.</li><li>Mechanism and Architecture</li><li>Traditional oracle networks have complex architectures that involve multiple components with different mechanisms. They also have many dependencies, such as complex tokenomics, and uncertain third-party reputation. In comparison, the zkOracle network has a much simpler architecture and its mechanism is much more straightforward. As long as there is one honest zkOracle node and the zk proofs are verified, all associated data is correct.</li></ul><h3>b) zkAutomation vs. Other Automation Protocols</h3><p>zkAutomation offers all the advantages of zkOracle network, along with specific benefits when compared to other automation protocols like Keep3r Network, Chainlink Automation, and Gelato Network. In addition to having all the features of these automation protocols, zkAutomation also provides:</p><ul><li>Trustless Off-chain Source</li><li>zkAutomation provides trustless automation based on off-chain source with programmable zkGraph. This allows for flexible automation that triggers based on off-chain computation results while maintaining a secure mechanism.</li><li>Verifiability and Security with ZKP</li><li>zkAutomation's zk proofs enable full verifiability. Other automation protocols usually rely on DAO, "social consensus", or even legal documents to restrict bad behaviors, which can create uncertainty and additional time or power burden for protecting the network's health. In contrast, zkAutomation's network is autonomous and automated with pure cryptography-based proof generation and verification without human intervention.</li></ul><h3>c) zkIndexing vs. Other Indexing Protocols</h3><p>zkIndexing offers all the benefits of the zkOracle network, as well as several advantages over other indexing protocols like The Graph. These include:</p><ul><li>Improved Performance with Trustlessness</li><li>Performance is a key metric for an indexing and querying service, with latency of request and response largely dependent on the geographic distance between the node and requester. Traditional decentralized indexing networks struggle with this due to the honest majority assumption. While requesters can trust the outcome from the entire network, they cannot trust a single node, which can affect performance. zkIndexing solves this with zk proofs. Developers can request data trustlessly from only one Hyper Oracle zkOracle node, which is geographically closest and fastest.</li><li>Verifiability and Security with ZKP</li><li>Traditional indexing networks currently only serve certain types of decentralized applications that do not consider data correctness as a critical component, such as dashboards. This is not due to a lack of desire on the part of developers to integrate these indexing services as core components, but rather because these networks are not yet secure enough through multi-sig controlled dispute council. zkIndexing addresses this issue with verifiability and security backed by the mathematics and cryptography of zk proofs. Developers can build any type of decentralized application with zkIndexing.</li></ul><h2>Summary</h2><p>This whitepaper introduces Hyper Oracle, a programmable zkOracle network that safeguard blockchain security and decentralization.</p><p>Hyper Oracle has three main components: zkPoS, zkGraph and zkWASM. zkPoS securely retrieves block headers and data roots from the Ethereum blockchain; zkWASM runs programmable customized off-chain computation and data mapping defined by zkGraph. Additionally, Hyper Oracle Meta Apps - zkAutomation and zkIndexing - provide secure automation for smart contracts and indexing/querying of blockchain data defined by zkGraph. Compared to other traditional oracle networks, Hyper Oracle addresses existing issues of security, decentralization, and performance through proposing a trustless zk powered infrastructure solution.</p><p>As the blockchain industry evolves and demand for the next generation of decentralized applications increases, Hyper Oracle, with advances in oracle technology, will serve as a crucial backbone for DApps and establish the new paradigm for programmable infrastructure.</p>
