edgar crawler
1.0.0
Edgar-Crawler是唯一從埃德加（Edgar）下載原始和非結構化的SEC文件，並將其轉換為結構化的JSON文件以進行Bootstrap Financial NLP實驗，並將其轉換為結構化的JSON文件。
Edgar-Crawler具有2個核心功能：
除了下載原始文件外， Edgar-Crawler是唯一將復雜且非結構化的SEC文件轉換為結構化JSON輸出的開源工具包，以便於您的研究和開發更容易集成。以下是每種支持的備案類型的此類輸出的示例：
原始報告：2022年的蘋果10-K
{
"cik" : " 320193 " ,
"company" : " Apple Inc. " ,
"filing_type" : " 10-K " ,
"filing_date" : " 2022-10-28 " ,
"period_of_report" : " 2022-09-24 " ,
"sic" : " 3571 " ,
"state_of_inc" : " CA " ,
"state_location" : " CA " ,
"fiscal_year_end" : " 0924 " ,
"filing_html_index" : " https://www.sec.gov/Archives/edgar/data/320193/0000320193-22-000108-index.html " ,
"htm_filing_link" : " https://www.sec.gov/Archives/edgar/data/320193/000032019322000108/aapl-20220924.htm " ,
"complete_text_filing_link" : " https://www.sec.gov/Archives/edgar/data/320193/0000320193-22-000108.txt " ,
"filename" : " 320193_10K_2022_0000320193-22-000108.htm " ,
"item_1" : " Item 1. Business n Company Background n The Company designs, manufactures ... " ,
"item_1A" : " Item 1A. Risk Factors n The Company’s business, reputation, results of ... " ,
"item_1B" : " Item 1B. Unresolved Staff Comments n None. " ,
"item_1C" : " " ,
"item_2" : " Item 2. Properties n The Company’s headquarters are located in Cupertino, California. ... " ,
"item_3" : " Item 3. Legal Proceedings n Epic Games n Epic Games, Inc. (“Epic”) filed a lawsuit ... " ,
"item_4" : " Item 4. Mine Safety Disclosures n Not applicable. ... " ,
"item_5" : " Item 5. Market for Registrant’s Common Equity, Related Stockholder ... " ,
"item_6" : " Item 6. [Reserved] n Apple Inc. | 2022 Form 10-K | 19 " ,
"item_7" : " Item 7. Management’s Discussion and Analysis of Financial Condition ... " ,
"item_8" : " Item 8. Financial Statements and Supplementary Data n All financial ... " ,
"item_9" : " Item 9. Changes in and Disagreements with Accountants on Accounting and Financial Disclosure n None. " ,
"item_9A" : " Item 9A. Controls and Procedures n Evaluation of Disclosure Controls and ... " ,
"item_9B" : " Item 9B. Other Information n Rule 10b5-1 Trading Plans n During the three months ... " ,
"item_9C" : " Item 9C. Disclosure Regarding Foreign Jurisdictions that Prevent Inspections n Not applicable. ... " ,
"item_10" : " Item 10. Directors, Executive Officers and Corporate Governance n The information required ... " ,
"item_11" : " Item 11. Executive Compensation n The information required by this Item will be included ... " ,
"item_12" : " Item 12. Security Ownership of Certain Beneficial Owners and Management and ... " ,
"item_13" : " Item 13. Certain Relationships and Related Transactions, and Director Independence ... " ,
"item_14" : " Item 14. Principal Accountant Fees and Services n The information required ... " ,
"item_15" : " Item 15. Exhibit and Financial Statement Schedules n (a)Documents filed as part ... " ,
"item_16" : " Item 16. Form 10-K Summary n None. n Apple Inc. | 2022 Form 10-K | 57 "
}原始報告：Apple 10-Q摘自第1季度2024
{
"cik" : " 320193 " ,
"company" : " Apple Inc. " ,
"filing_type" : " 10-Q " ,
"filing_date" : " 2024-05-03 " ,
"period_of_report" : " 2024-03-30 " ,
"sic" : " 3571 " ,
"state_of_inc" : " CA " ,
"state_location" : " CA " ,
"fiscal_year_end" : " 0928 " ,
"filing_html_index" : " https://www.sec.gov/Archives/edgar/data/320193/0000320193-24-000069-index.html " ,
"htm_filing_link" : " https://www.sec.gov/Archives/edgar/data/320193/000032019324000069/aapl-20240330.htm " ,
"complete_text_filing_link" : " https://www.sec.gov/Archives/edgar/data/320193/0000320193-24-000069.txt " ,
"filename" : " 320193_10Q_2024_0000320193-24-000069.htm " ,
"part_1" : " PART I - FINANCIAL INFORMATION n Item 1. Financial Statements n Apple Inc. n CONDENSED CONSOLIDATED STATEMENTS ... " ,
"part_1_item_1" : " Item 1. Financial Statements n Apple Inc. n CONDENSED CONSOLIDATED STATEMENTS ... " ,
"part_1_item_2" : " Item 2. Management’s Discussion and Analysis of Financial Condition and ... " ,
"part_1_item_3" : " Item 3. Quantitative and Qualitative Disclosures About Market Risk n There have ... " ,
"part_1_item_4" : " Item 4. Controls and Procedures n Evaluation of Disclosure Controls and ... " ,
"part_2" : " PART II - OTHER INFORMATION n Item 1. Legal Proceedings n Digital Markets Act Investigations n On ... " ,
"part_2_item_1" : " Item 1. Legal Proceedings n Digital Markets Act Investigations n On March 25, 2024, ... " ,
"part_2_item_1A" : " Item 1A. Risk Factors n The Company’s business, reputation, ... " ,
"part_2_item_2" : " Item 2. Unregistered Sales of Equity Securities and Use of ... " ,
"part_2_item_3" : " Item 3. Defaults Upon Senior Securities n None. " ,
"part_2_item_4" : " Item 4. Mine Safety Disclosures n Not applicable. " ,
"part_2_item_5" : " Item 5. Other Information n Insider Trading Arrangements n None. " ,
"part_2_item_6" : " Item 6. Exhibits n Incorporated by Reference n Exhibit n Number n Exhibit Description ... "
}注意： part_1和part_2包含該部分的完整檢測到的文本。我們提供了這一點，因為在某些舊的10-Q文件中，不可能在項目級別提取信息。
原始報告：2022-08-19的Apple 8-K
{
"cik" : " 320193 " ,
"company" : " Apple Inc. " ,
"filing_type" : " 8-K " ,
"filing_date" : " 2022-08-19 " ,
"period_of_report" : " 2022-08-17 " ,
"sic" : " 3571 " ,
"state_of_inc" : " CA " ,
"state_location" : " CA " ,
"fiscal_year_end" : " 0924 " ,
"filing_html_index" : " https://www.sec.gov/Archives/edgar/data/320193/0001193125-22-225365-index.html " ,
"htm_filing_link" : " https://www.sec.gov/Archives/edgar/data/320193/000119312522225365/d366128d8k.htm " ,
"complete_text_filing_link" : " https://www.sec.gov/Archives/edgar/data/320193/0001193125-22-225365.txt " ,
"filename" : " 320193_8K_2022_0001193125-22-225365.htm " ,
"item_1.01" : " " ,
"item_1.02" : " " ,
"item_1.03" : " " ,
"item_1.04" : " " ,
"item_1.05" : " " ,
"item_2.01" : " " ,
"item_2.02" : " " ,
"item_2.03" : " " ,
"item_2.04" : " " ,
"item_2.05" : " " ,
"item_2.06" : " " ,
"item_3.01" : " " ,
"item_3.02" : " " ,
"item_3.03" : " " ,
"item_4.01" : " " ,
"item_4.02" : " " ,
"item_5.01" : " " ,
"item_5.02" : " Item 5.02 Departure of Directors or Certain Officers; Election of Directors; Appointment ... " ,
"item_5.03" : " Item 5.03 Amendments to Articles of Incorporation or Bylaws; Change in Fiscal Year. n On August 17, 2022, Apple’s Board approved and adopted amended and restated bylaws ... " ,
"item_5.04" : " " ,
"item_5.05" : " " ,
"item_5.06" : " " ,
"item_5.07" : " " ,
"item_5.08" : " " ,
"item_6.01" : " " ,
"item_6.02" : " " ,
"item_6.03" : " " ,
"item_6.04" : " " ,
"item_6.05" : " " ,
"item_7.01" : " " ,
"item_8.01" : " " ,
"item_9.01" : " Item 9.01 Financial Statements and Exhibits. n (d) Exhibits. n Exhibit n Number n Exhibit ... " ,
}EDGAR-CRAWLER ： # Method 1: SSH
git clone https://github.com/nlpaueb/edgar-crawler.git
# Method 2: HTTPS
git clone [email protected]:nlpaueb/edgar-crawler.gitconda create -n edgar-crawler-venv python=3.8 # After installing Anaconda, create a venv with python 3.8+
conda activate edgar-crawler-venv # Activate the environmentpip install -r requirements.txt # Install requirements for edgar-crawler 在運行任何腳本之前，您應該編輯config.json文件，該文件配置了我們的2個模塊的行為（一個用於下載您選擇的文件，另一個用於獲取其結構化輸出）。
download_filings.py的論點，下載財務報告的模塊：start_year XXXX ：開始的年度範圍（默認為2023）。end_year YYYY ：結束的年度範圍（默認值為2023）。quarters ：您要從（列表）下載文件的四分之一。[1, 2, 3, 4] 。filing_types ：要下載的備案類型列表。['10-K', '8-K', '10-Q'] 。cik_tickers ：包含CIK或訴訟的文件的列表或路徑。例如[789019, "1018724", "AAPL", "TWTR"]user_agent ：將聲明為SEC EDGAR的用戶代理（名稱/電子郵件）。raw_filings_folder ：將存儲下載文件的文件夾的名稱。'RAW_FILINGS' 。indices_folder ：將存儲Edgar TSV文件的文件夾的名稱。這些用於定位年度報告。默認值是'INDICES' 。filings_metadata_file ：CSV文件名從報告中保存元數據。skip_present_indices ：是否跳過已經下載了Edgar索引或下載它們。True 。extract_items.py的參數，從已經下載的報告中清潔和提取文本數據的模塊：raw_filings_folder ：存儲下載文檔的文件夾的名稱。'RAW_FILINGS' 。extracted_filings_folder ：將存儲提取文檔的文件夾的名稱。'EXTRACTED_FILINGS' 。filings_metadata_file ：加載報告元數據的CSV文件名（提供與download_filings.py中相同的CSV文件）。filing_types ：要提取的備案類型列表。include_signature ：是否在最後一項之後包括簽名部分。items_to_extract ：帶有特定項目部分的列表。['7','8']提取“管理層的討論和分析”和“財務報表”部分項目的10-K報告。remove_tables ：是否要刪除包含數值（財務）數據的表。這項工作主要是為了促進NLP研究，而數值表通常沒有用。skip_extracted_filings ：是否跳過已提取文件或提取它們。True 。要從Edgar下載原始財務報告，請運行python download_filings.py 。
要從已經下載的文檔中清理和提取特定的項目部分，請運行python extract_items.py 。
part都包含在單獨的條目中。 Edgar-Crawler紙正在途中。在此之前，請引用我們在Econlp@emnlp 2021（多米尼加共和國Punta Cana）上發表的相關埃德加·科普斯論文。
@inproceedings { loukas-etal-2021-edgar-corpus-and-edgar-crawler ,
title = " {EDGAR}-{CORPUS}: {B}illions of {T}okens {M}ake {T}he {W}orld {G}o {R}ound " ,
author = " Loukas, Lefteris and
Fergadiotis, Manos and
Androutsopoulos, Ion and
Malakasiotis, Prodromos " ,
booktitle = " Proceedings of the Third Workshop on Economics and Natural Language Processing (ECONLP) " ,
month = nov,
year = " 2021 " ,
address = " Punta Cana, Dominican Republic " ,
publisher = " Association for Computational Linguistics " ,
url = " https://aclanthology.org/2021.econlp-1.2 " ,
pages = " 13--18 " ,
}在此處閱讀Edgar-Corpus論文：https：//aclanthology.org/2021.econlp-1.2/
以下是使用Edgar-Crawler創建的一些其他資源：
Edgar-Corpus ：最大的金融NLP語料庫，年度報告中有6億個令牌（Huggingface URL？）| （Zenodo URL）。
EDGAR-W2V ：財務Word2Vec嵌入，預先訓練在Edgar-Corpus（Zenodo url）上
您有任何功能請求嗎？直接使用此Google表單告訴我們：（https：//forms.gle/bpv8nxmqx8sq2v5z8）！
公關和貢獻被接受。我們使用功能分支工作流程。
請在Github上創建問題，而不是直接向我們發送電子郵件，以便所有可能的用戶都可以從故障排除中受益。
該軟件是根據GNU通用公共許可證v3.0許可的，該許可是由開源計劃（OSI）批准的許可證。
