Skip to content

hakmarkdev/data-ingestions

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

5 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Data Ingestions Workspace

A collection of high-performance blockchain data ingestion services (producers) that fetch data from diversas sources and publish it to Kafka using shared Protocol Buffer definitions.

Services

Directory Language Source Target Chain(s)
eth-go-producers Go JSON-RPC Ethereum
evm-producers TypeScript JSON-RPC EVM-Compatible (Base, HyperEVM, etc.)
hyperliquid-producers TypeScript WebSocket Hyperliquid L1
solana-producers TypeScript gRPC (Yellowstone) Solana

Overview

Each service in this workspace is responsible for:

  1. Connecting to a specific blockchain data source (RPC, WebSocket, or gRPC).
  2. Mapping raw blockchain data to shared Protobuf structures.
  3. Publishing the structured data to Kafka topics for downstream consumption.

Shared Dependencies

Producers rely on the following shared packages (located in sibling repositories or managed via workspace links):

  • @ethereum-kafka-indexer/protos: Common Protocol Buffer definitions.
  • @ethereum-kafka-indexer/shared: Shared configuration and utility logic.

About

Data Ingestions Workspace

Topics

Resources

Stars

Watchers

Forks

Contributors

Languages