dbt artifacts parser

DBT-Artifacts-Parser

Это разбор артефактов DBT в Python. Это позволяет нам иметь дело с catalog.json , manifest.json , run-results.json и sources.json .

Этот пакет в основном предназначен для DBT-Core, что позволяет бесшовно взаимодействовать с артефактами DBT в качестве объектов Python. В то время как DBT Cloud предоставляет дополнительные типы артефактов, помимо DBT-CORE, этот пакет не предлагает им всестороннюю поддержку.

Поддерживаемые версии и совместимость

️ Важное примечание:

• Pydantic V1 не будет поддерживаться для DBT 1,9 или более поздней версии.
• Для анализа DBT 1.9 или более поздней версии, пожалуйста, перенести свой код в Pydantic V2.
• Мы переоценим совместимость с версией после выпуска Pydantic V3.

Версия	Поддерживаемая версия DBT	Поддерживаемая пиджантическая версия
0,8	DBT от 1,5 до 1,9	Pydantic V2
0,7	DBT от 1,5 до 1,8	Pydantic V2
0,6	DBT от 1,5 до 1,8	Pydantic V1
0,5	DBT от 1,5 до 1,7	Pydantic V1

Установка

pip install -U dbt-artifacts-parser

Классы Python

Это классы для разбора артефактов DBT.

Каталог

• Catalogv1 для catalog.json v1

Манифест

• Manifestv1 для manifest.json v1
• Manifestv2 для manifest.json v2
• Manifestv3 для manifest.json v3
• Manifestv4 для manifest.json v4
• Manifestv5 для manifest.json v5
• Manifestv6 для manifest.json v6
• Manifestv7 для manifest.json v7
• Manifestv8 для manifest.json v8
• Manifestv9 для manifest.json v9
• Manifestv10 для manifest.json v10
• Manifestv11 для manifest.json v11
• Manifestv12 для manifest.json v12

Запустить результаты

• Runresultsv1 для run_results.json v1
• Runresultsv2 для run_results.json v2
• Runresultsv3 для run_results.json v3
• Runresultsv4 для run_results.json v4
• Runresultsv5 для run_results.json v5
• Runresultsv6 для run_results.json v6

Источники

• Источники для источников.json v1
• Источники для источников. JSON V2
• Источники для источников.json v3

Примеры

Parse Catalog.json

 import json

# parse any version of catalog.json
from dbt_artifacts_parser . parser import parse_catalog

with open ( "path/to/catalog.json" , "r" ) as fp :
    catalog_dict = json . load ( fp )
    catalog_obj = parse_catalog ( catalog = catalog_dict )

# parse catalog.json v1
from dbt_artifacts_parser . parser import parse_catalog_v1

with open ( "path/to/catalog.json" , "r" ) as fp :
    catalog_dict = json . load ( fp )
    catalog_obj = parse_catalog_v1 ( catalog = catalog_dict )

Parse manifest.json

 import json

# parse any version of manifest.json
from dbt_artifacts_parser . parser import parse_manifest

with open ( "path/to/manifest.json" , "r" ) as fp :
    manifest_dict = json . load ( fp )
    manifest_obj = parse_manifest ( manifest = manifest_dict )

# parse manifest.json v1
from dbt_artifacts_parser . parser import parse_manifest_v1

with open ( "path/to/manifest.json" , "r" ) as fp :
    manifest_dict = json . load ( fp )
    manifest_obj = parse_manifest_v1 ( manifest = manifest_dict )

# parse manifest.json v2
from dbt_artifacts_parser . parser import parse_manifest_v2

with open ( "path/to/manifest.json" , "r" ) as fp :
    manifest_dict = json . load ( fp )
    manifest_obj = parse_manifest_v2 ( manifest = manifest_dict )

# parse manifest.json v3
from dbt_artifacts_parser . parser import parse_manifest_v3

with open ( "path/to/manifest.json" , "r" ) as fp :
    manifest_dict = json . load ( fp )
    manifest_obj = parse_manifest_v3 ( manifest = manifest_dict )

# parse manifest.json v4
from dbt_artifacts_parser . parser import parse_manifest_v4

with open ( "path/to/manifest.json" , "r" ) as fp :
    manifest_dict = json . load ( fp )
    manifest_obj = parse_manifest_v4 ( manifest = manifest_dict )

# parse manifest.json v5
from dbt_artifacts_parser . parser import parse_manifest_v5

with open ( "path/to/manifest.json" , "r" ) as fp :
    manifest_dict = json . load ( fp )
    manifest_obj = parse_manifest_v5 ( manifest = manifest_dict )

# parse manifest.json v6
from dbt_artifacts_parser . parser import parse_manifest_v6

with open ( "path/to/manifest.json" , "r" ) as fp :
    manifest_dict = json . load ( fp )
    manifest_obj = parse_manifest_v6 ( manifest = manifest_dict )

# parse manifest.json v7
from dbt_artifacts_parser . parser import parse_manifest_v7

with open ( "path/to/manifest.json" , "r" ) as fp :
    manifest_dict = json . load ( fp )
    manifest_obj = parse_manifest_v7 ( manifest = manifest_dict )

# parse manifest.json v8
from dbt_artifacts_parser . parser import parse_manifest_v8

with open ( "path/to/manifest.json" , "r" ) as fp :
    manifest_dict = json . load ( fp )
    manifest_obj = parse_manifest_v8 ( manifest = manifest_dict )

# parse manifest.json v9
from dbt_artifacts_parser . parser import parse_manifest_v9

with open ( "path/to/manifest.json" , "r" ) as fp :
    manifest_dict = json . load ( fp )
    manifest_obj = parse_manifest_v9 ( manifest = manifest_dict )

# parse manifest.json v10
from dbt_artifacts_parser . parser import parse_manifest_v10

with open ( "path/to/manifest.json" , "r" ) as fp :
    manifest_dict = json . load ( fp )
    manifest_obj = parse_manifest_v10 ( manifest = manifest_dict )

# parse manifest.json v11
from dbt_artifacts_parser . parser import parse_manifest_v11

with open ( "path/to/manifest.json" , "r" ) as fp :
    manifest_dict = json . load ( fp )
    manifest_obj = parse_manifest_v11 ( manifest = manifest_dict )

# parse manifest.json v12
from dbt_artifacts_parser . parser import parse_manifest_v12

with open ( "path/to/manifest.json" , "r" ) as fp :
    manifest_dict = json . load ( fp )
    manifest_obj = parse_manifest_v12 ( manifest = manifest_dict )

Parse Run-Results.json

 import json

# parse any version of run-results.json
from dbt_artifacts_parser . parser import parse_run_results

with open ( "path/to/run-resultsjson" , "r" ) as fp :
    run_results_dict = json . load ( fp )
    run_results_obj = parse_run_results ( run_results = run_results_dict )

# parse run-results.json v1
from dbt_artifacts_parser . parser import parse_run_results_v1

with open ( "path/to/run-results.json" , "r" ) as fp :
    run_results_dict = json . load ( fp )
    run_results_obj = parse_run_results_v1 ( run_results = run_results_dict )

# parse run-results.json v2
from dbt_artifacts_parser . parser import parse_run_results_v2

with open ( "path/to/run-results.json" , "r" ) as fp :
    run_results_dict = json . load ( fp )
    run_results_obj = parse_run_results_v2 ( run_results = run_results_dict )

# parse run-results.json v3
from dbt_artifacts_parser . parser import parse_run_results_v3

with open ( "path/to/run-results.json" , "r" ) as fp :
    run_results_dict = json . load ( fp )
    run_results_obj = parse_run_results_v3 ( run_results = run_results_dict )

# parse run-results.json v4
from dbt_artifacts_parser . parser import parse_run_results_v4

with open ( "path/to/run-results.json" , "r" ) as fp :
    run_results_dict = json . load ( fp )
    run_results_obj = parse_run_results_v4 ( run_results = run_results_dict )

# parse run-results.json v5
from dbt_artifacts_parser . parser import parse_run_results_v5

with open ( "path/to/run-results.json" , "r" ) as fp :
    run_results_dict = json . load ( fp )
    run_results_obj = parse_run_results_v5 ( run_results = run_results_dict )

# parse run-results.json v6
from dbt_artifacts_parser . parser import parse_run_results_v6

with open ( "path/to/run-results.json" , "r" ) as fp :
    run_results_dict = json . load ( fp )
    run_results_obj = parse_run_results_v6 ( run_results = run_results_dict )

Parse Sources.json

 import json

# parse any version of sources.json
from dbt_artifacts_parser . parser import parse_sources

with open ( "path/to/sources.json" , "r" ) as fp :
    sources_dict = json . load ( fp )
    sources_obj = parse_sources ( sources = sources_dict )

# parse sources.json v1
from dbt_artifacts_parser . parser import parse_sources_v1

with open ( "path/to/sources.json" , "r" ) as fp :
    sources_dict = json . load ( fp )
    sources_obj = parse_sources_v1 ( sources = sources_dict )

# parse sources.json v2
from dbt_artifacts_parser . parser import parse_sources_v2

with open ( "path/to/sources.json" , "r" ) as fp :
    sources_dict = json . load ( fp )
    sources_obj = parse_sources_v2 ( sources = sources_dict )

# parse sources.json v3
from dbt_artifacts_parser . parser import parse_sources_v3

with open ( "path/to/sources.json" , "r" ) as fp :
    sources_dict = json . load ( fp )
    sources_obj = parse_sources_v3 ( sources = sources_dict )

Участники

Спасибо за ваш вклад!

Если вы заинтересованы в участии в этом пакете, ознакомьтесь с Anforming.md.

Юикава	Нулевой	Брент Брюрингтон	Иуда Рэнд	МАЙУР СИНГАЛ	Мохамед Набил Махмуд Хафез
Онкар Равган	Нулевой