desafio2 prof artemisia

Challenge2-Prof-Artemisia

Description:

This project was developed as part of the mentor I get. The goal is to load the cloud data after being properly treated so that they are accessible to BI tools.

Features:

• Data processing from a .csv format dataset
• Data transformation with cleaning, standardization and enrichment in SQL, initially in a local bank (Postgres).
• Creation of a Datalake at Cloud, with layers: Raw, Silver, Gold and Diamond.
• Consumption of data via BI tool.

Installation:

1. Clone this repository:

   git clone https://github.com/DaviMacielCavalcante/desafio2-prof-artemisia
   cd desafio2-prof-artemisia

2. Download the indexData.csv file of this link https://www.kaggle.com/datasets/mattiuzc/stock-exchange-data
3. At the root of the project, create a directory called "Datasets", and put the indexData.csv file in it.

• I recommend cleaning the .csv files present in the Datalake layers, just to have the experience of seeing everything happens or change the scripts the way you prefer.

4. Run the scrip that is responsible for creating the Silver layer:

   python preparando_camada_silver.py

5. Then the Gold layer:

 python preparando_camada_gold.py

7. Finally, the diamond layer:

 python preparando_camada_diamond.py

8. Climbing to the cloud:
   • Create an AWS account;
   • Follow this tutorial at AWS LATAM to climb Datalake:

   https://youtube.com/playlist ? list=PLQHh55hXC4yrBZ4yookmQPlX2zM9dZ-MH & si=lpGE6Hz2F6t37THw

   • If you want to connect to Power BI, follow this tutorial:

    https://youtu.be/WS3LUbK0ung ? si=YXc_Wy5j53Ct34z3

9. Continue on the legal path of force:

How to contribute:

Contributions are welcome! Please follow these guidelines:

• Take a project fork.
• Create a branch for the functionality you want to implement (git checkout -b my-nova-feature).
• Make the Commits with good descriptions (Git Commit -m 'adds new feature').
• Make a push to the created branch (Git push origin my-new-feature).
• Open a Request Pull for Review.

License

This project is licensed under the MIT license - see the license.md file for more details.

Contact

If you have any questions or problems, please contact us:

E-mail: [email protected]

Lordes Sith responsible for the project

• Darth David ⚔️?

Mentor that proposed the challenge:

Profa. Artemisia Weyl

Mentor Github: https://github.com/arteweyl

Through Victory, My Chains Are Broken.
The Force Shall Free Me.