e2e

0xgabey

v0.4.0/14 downloads

Package ref

0xgabey/e2e@v0.4.0

Download .spkg

Run package

Start block: latest 200 blocks

Deploy it

Modules

Execution graph

9 modules

mapMain

map_events_0

Output

test.output.Events

Inputs

sf.acme.type.v1.Block

map

map_downstream

from #100

Output

test.output.Events

Inputs

map_filtered

map

map_events

from #100

Output

test.output.Events

Inputs

sf.acme.type.v1.Block

map

map_filtered

from #100

Output

test.output.Events

Inputs

map_events

map

map_hosted_store_feed

Output

sf.substreams.foundational_store.model.v2.SinkEntries

Inputs

sf.acme.type.v1.Block

map

map_stats

Output

test.output.Events

Inputs

sf.acme.type.v1.Block store_stats

map

map_stats2

from #150

Output

test.output.Events

Inputs

sf.acme.type.v1.Block store_stats2

store

store_stats

from #400

Store value

bigint

Update policy

add

Inputs

sf.acme.type.v1.Block

store

store_stats2

from #150

Store value

bigint

Update policy

add

Inputs

sf.acme.type.v1.Block store_stats

Networks

dummydefault

The manifest declares this network only.

Protobuf

test.output

Eventsmain

test.output.Events

Event[]event = 1

Event

test.output.Event

Annotated for the ClickHouse sink, which has no default ordering and refuses to create a table without one. Postgres ignores these options.

stringevt_tx_hash = 1

ClickHouse requires the primary key to be a prefix of the sorting key.

stringevt_from = 2

stringevt_to = 3

uint64evt_block_number = 4

uint64fee = 5

StoreQuery

test.output.StoreQuery

StoreQuery is what the two consumer modules emit: the block they ran on, plus one result per key they looked up.

uint64block_number = 1

StoreQueryEntry[]entry = 2

StoreQueryEntry

test.output.StoreQueryEntry

stringkey = 1

stringcode = 2

Name of the sf.substreams.foundational_store.model.v2.ResponseCode the store answered with, so a miss stays visibly distinct from a decode failure.

stringtype_url = 3

Type URL the entry's Any carried, kept so a Type URL mismatch is visible.

StoreValuevalue = 4

Set only when the entry was found and decoded as a StoreValue.

StoreValue

test.output.StoreValue

StoreValue is the value type map_hosted_store_feed writes into a Hosted Store. It is carried inside a google.protobuf.Any, so its fully-qualified name -- test.output.StoreValue -- is what you give as the store's "Type URL" when creating the service.

uint64block_number = 1

stringblock_hash = 2

uint64transaction_count = 3

Show 33 dependencies

schema

ClickhouseIndexField

schema.ClickhouseIndexField

ClickhouseIndexField defines a secondary index for improved query performance. Secondary indexes help ClickHouse skip irrelevant data blocks during query execution.

stringname = 1

The name of the index (must be unique within the table)

stringfield_name = 2

The field name on which to create the index

IndexTypetype = 3

The type of index to create (e.g., minmax, bloom_filter, set)

uint32granularity = 4

The granularity parameter that controls index density (higher = less memory, lower precision)

Functionfunction = 5

Optional function to apply to the field value before indexing

ClickhouseOrderByField

schema.ClickhouseOrderByField

ClickhouseOrderByField defines a field used in the ClickHouse table ORDER BY clause. ORDER BY determines the physical storage order of data for optimal compression and query performance.

stringname = 1

The name of the field to include in the ORDER BY clause

booldescending = 2

Whether to sort this field in descending order (default is ascending)

Functionfunction = 3

Optional function to apply to the field value before ordering (e.g., toDate for timestamps)

ClickhousePartitionByField

schema.ClickhousePartitionByField

ClickhousePartitionByField defines a field used for ClickHouse table partitioning. Partitioning splits data into separate parts based on the specified field values.

stringname = 1

The name of the field to use for partitioning

Functionfunction = 2

Optional function to apply to the field value before partitioning (e.g., toYYYYMM for dates)

ClickhouseTableOptions

schema.ClickhouseTableOptions

ClickhouseTableOptions defines ClickHouse-specific table configuration options. These options are used to configure ClickHouse table engines and optimize query performance.

ClickhouseOrderByField[]order_by_fields = 1

Fields to use for the ORDER BY clause in ClickHouse table definition. ORDER BY determines the physical ordering of data on disk for better compression and query performance.

ClickhousePartitionByField[]partition_fields = 2

Fields to use for table partitioning in ClickHouse. Partitioning helps with query performance and data management by splitting data into logical parts.

ClickhouseIndexField[]index_fields = 4

Secondary indexes to create for improved query performance on specific columns. These are data skipping indexes that help ClickHouse skip irrelevant data blocks during queries.

Column

schema.Column

Column defines metadata for mapping a protobuf field to a database column. This message is used with the (field) extension on protobuf fields.

stringname = 1

Custom column name. If not specified, the field name is used.

stringforeign_key = 2

Foreign key reference to another table's column in the format "table.column". Used to establish referential integrity constraints in relational databases.

boolunique = 3

Whether this column should have a unique constraint. Ensures no duplicate values exist for this column.

boolprimary_key = 4

Whether this column is part of the primary key. Primary key columns are automatically non-null and unique.

StringConvertionconvertTo = 5

Specifies how to convert string fields to specific numeric types. Only applies to string fields that contain numeric data. Leave unset for no conversion (store as string).

boolinline = 6

Decimal128

schema.Decimal128

Decimal128 represents a 128-bit decimal type for string conversion. Used in StringConvertion to specify that a string field should be converted to Decimal128.

int32scale = 1

The number of digits after the decimal point for this decimal type.

Decimal256

schema.Decimal256

Decimal256 represents a 256-bit decimal type for string conversion. Used in StringConvertion to specify that a string field should be converted to Decimal256.

int32scale = 1

The number of digits after the decimal point for this decimal type.

Int128

schema.Int128

Int128 represents a 128-bit signed integer type for string conversion. Used in StringConvertion to specify that a string field should be converted to Int128.

No fields.

Int256

schema.Int256

Int256 represents a 256-bit signed integer type for string conversion. Used in StringConvertion to specify that a string field should be converted to Int256.

No fields.

StringConvertion

schema.StringConvertion

StringConvertion defines how string fields should be converted to specific numeric types. This is used when protobuf string fields contain numeric data that should be stored as native numeric types in the database for better performance and storage efficiency. Usage: Set this in Column.convertTo field to specify the target numeric type.

Int128int128 = 2oneof convertion

Convert string to 128-bit signed integer

UInt128uint128 = 3oneof convertion

Convert string to 128-bit unsigned integer

Int256int256 = 4oneof convertion

Convert string to 256-bit signed integer

UInt256uint256 = 6oneof convertion

Convert string to 256-bit unsigned integer

Decimal128decimal128 = 50oneof convertion

Convert string to 128-bit decimal with specified scale

Decimal256decimal256 = 51oneof convertion

Convert string to 256-bit decimal with specified scale

Table

schema.Table

Table defines metadata for mapping a protobuf message to a database table. This message is used with the (table) extension on protobuf messages.

stringname = 1

The name of the database table. If not specified, the message name is used.

stringchild_of = 2

Optional field to specify a parent table for establishing table relationships. Used for creating foreign key relationships in relational databases.

ClickhouseTableOptionsclickhouse_table_options = 200

ClickHouse-specific table options for configuring table engine parameters. Only applicable when using ClickHouse as the target database.

UInt128

schema.UInt128

UInt128 represents a 128-bit unsigned integer type for string conversion. Used in StringConvertion to specify that a string field should be converted to UInt128.

No fields.

UInt256

schema.UInt256

UInt256 represents a 256-bit unsigned integer type for string conversion. Used in StringConvertion to specify that a string field should be converted to UInt256.

No fields.

Functionenum

schema.Function

unset = 0

No function applied (use raw field value)

toYYYYMM = 1

Convert date/timestamp to YYYYMM format (useful for monthly partitioning)

toYYYYDD = 2

Convert date/timestamp to YYYYDD format (useful for daily partitioning)

toYear = 3

Extract year from date/timestamp (useful for yearly partitioning)

toMonth = 4

Extract month from date/timestamp (useful for monthly grouping)

toDate = 5

Convert timestamp to date (removes time component)

toStartOfMonth = 6

Convert date to first day of the month (useful for monthly aggregations)

IndexTypeenum

schema.IndexType

minmax = 0

MinMax index stores minimum and maximum values for each granule (fastest, least memory)

set = 1

Set index stores unique values for each granule (good for low cardinality fields)

ngrambf_v1 = 2

N-gram Bloom filter index for substring searches (good for text search)

tokenbf_v1 = 3

Token Bloom filter index for full-word searches (good for tokenized text)

bloom_filter = 4

Bloom filter index for equality checks (good for high cardinality fields)

sf.acme.type.v1

Attribute

sf.acme.type.v1.Attribute

stringkey = 1

stringvalue = 2

BigInt

sf.acme.type.v1.BigInt

bytesbytes = 1

Block

sf.acme.type.v1.Block

BlockHeaderheader = 1

Transaction[]transactions = 2

BlockHeader

sf.acme.type.v1.BlockHeader

uint64height = 1

stringhash = 2

uint64previous_num = 3

stringprevious_hash = 4

uint64final_num = 5

stringfinal_hash = 6

int64timestamp = 7

Event

sf.acme.type.v1.Event

stringtype = 1

Attribute[]attributes = 2

Transaction

sf.acme.type.v1.Transaction

stringtype = 1

stringhash = 2

stringsender = 3

stringreceiver = 4

BigIntamount = 5

BigIntfee = 6

boolsuccess = 7

Event[]events = 8

sf.firehose.v2

BlockMetadata

sf.firehose.v2.BlockMetadata

uint64num = 1

Num is the block number of this response's block.

stringid = 2

ID is the block ID of this response's block. The ID actual representation is chain specific.

Antelope & Ethereum uses hex.
NEAR & Solana uses base58.

Refer to the chain documentation for more details.

uint64parent_num = 3

ParentNum is the block number of the parent of this response's block

stringparent_id = 4

ParentID is the block ID of the parent of this response's block. If this response is the genesis block, this field is empty.

The ID actual representation is chain specific.

Antelope & Ethereum uses hex.
NEAR & Solana uses base58.

Refer to the chain documentation for more details.

uint64lib_num = 5

LibNum is the block number of the last irreversible block (a.k.a last finalized block) at the time of this response' block. It determines the finality of the block.

Timestamptime = 6

Time is the time at which the block was produced.

InfoRequest

sf.firehose.v2.InfoRequest

No fields.

InfoResponse

sf.firehose.v2.InfoResponse

stringchain_name = 1

Canonical chain name from https://thegraph.com/docs/en/developing/supported-networks/ (ex: matic, mainnet ...)

string[]chain_name_aliases = 2

Alternate names for the chain.

uint64first_streamable_block_num = 3

First block that is served by this endpoint. This should usually be the genesis block, but some providers may have truncated history.

stringfirst_streamable_block_id = 4

InfoResponse.BlockIdEncodingblock_id_encoding = 5

This informs the client on how to decode the block_id field inside the "Clock" message as well as the first_streamable_block_id above.

string[]block_features = 10

features describes the blocks. Popular values for EVM chains include base, extended or hybrid.

Request

sf.firehose.v2.Request

int64start_block_num = 1

Controls where the stream of blocks will start.

The stream will start inclusively at the requested block num.

When not provided, starts at first streamable block of the chain. Not all chain starts at the same block number, so you might get an higher block than requested when using default value of 0.

Can be negative, will be resolved relative to the chain head block, assuming a chain at head block #100, then using -50 as the value will start at block #50. If it resolves before first streamable block of chain, we assume start of chain.

If start_cursor is given, this value is ignored and the stream instead starts immediately after the Block pointed by the opaque start_cursor value.

stringcursor = 2

Controls where the stream of blocks will start which will be immediately after the Block pointed by this opaque cursor.

Obtain this value from a previously received Response.cursor.

This value takes precedence over start_block_num.

uint64stop_block_num = 3

When non-zero, controls where the stream of blocks will stop.

The stream will close after that block has passed so the boundary is inclusive.

boolfinal_blocks_only = 4

With final_block_only, you only receive blocks with STEP_FINAL Default behavior will send blocks as STEP_NEW, with occasional STEP_UNDO

Any[]transforms = 10

Response

sf.firehose.v2.Response

Anyblock = 1

Chain specific block payload, ex:

sf.eosio.type.v1.Block
sf.ethereum.type.v1.Block
sf.near.type.v1.Block

ForkStepstep = 6

stringcursor = 10

BlockMetadatametadata = 12

Metadata about the block, added in some Firehose version, so consumer should be ready to handle the absence of this field.

SingleBlockRequest

sf.firehose.v2.SingleBlockRequest

SingleBlockRequest.BlockNumberblock_number = 3oneof reference

SingleBlockRequest.BlockHashAndNumberblock_hash_and_number = 4oneof reference

SingleBlockRequest.Cursorcursor = 5oneof reference

Any[]transforms = 6

BlockHashAndNumber

sf.firehose.v2.SingleBlockRequest.BlockHashAndNumber

Get the current block with specific hash and number

uint64num = 1

stringhash = 2

BlockNumber

sf.firehose.v2.SingleBlockRequest.BlockNumber

Get the current known canonical version of a block at with this number

uint64num = 1

Cursor

sf.firehose.v2.SingleBlockRequest.Cursor

Get the block that generated a specific cursor

stringcursor = 1

SingleBlockResponse

sf.firehose.v2.SingleBlockResponse

Anyblock = 1

BlockMetadatametadata = 2

Metadata about the block, added in some Firehose version, so consumer should be ready to handle the absence of this field.

ForkStepenum

sf.firehose.v2.ForkStep

STEP_UNSET = 0

STEP_NEW = 1

Incoming block

STEP_UNDO = 2

A reorg caused this specific block to be excluded from the chain

STEP_FINAL = 3

Block is now final and can be committed (finality is chain specific, see chain documentation for more details)

BlockIdEncodingenum

sf.firehose.v2.InfoResponse.BlockIdEncoding

BLOCK_ID_ENCODING_UNSET = 0

BLOCK_ID_ENCODING_HEX = 1

BLOCK_ID_ENCODING_0X_HEX = 2

BLOCK_ID_ENCODING_BASE58 = 3

BLOCK_ID_ENCODING_BASE64 = 4

BLOCK_ID_ENCODING_BASE64URL = 5