Ubuntu,Macos和Windows:Windows:每週:
JSON解析器可能會在以後的版本中更改解析聯合向量的接口,該版本需要代碼生成匹配庫版本。
flatcc除構建和編譯器工具以及C運行時庫外沒有外部依賴關係。 With concurrent Ninja builds, a small client project can build flatcc with libraries, generate schema code, link the project and execute a test case in a few seconds, produce binaries between 15K and 60K, read small buffers in 30ns, build FlatBuffers in about 600ns, and with a larger executable also handle optional json parsing or printing in less than 2 us for a 10 field mixed type message.
該項目構建FlatCC,這是一個編譯器,該編譯器為C flatbuffer模式文件生成Flatbuffers代碼。本簡介還使用傳統的怪物示例創建了一個單獨的測試項目,此處在C版本中。
現在,假設一個類似UNIX的系統,儘管這不是一般要求 - 另請參見構建。您將需要git,cmake,bash,a C編譯器以及忍者構建系統或製造。
git clone https://github.com/dvidelabs/flatcc.git
cd flatcc
# scripts/initbuild.sh ninja
scripts/initbuild.sh make
scripts/setup.sh -a ../mymonster
ls bin
ls lib
cd ../mymonster
ls src
scripts/build.sh
ls generated
scripts/initbuild.sh是可選的,並且選擇了構建後端,該後端默認為忍者。
設置腳本使用CMAKE構建FlatCC,然後使用Monster示例創建一個測試項目目錄,而構建腳本只是一個小型的Shell腳本。標題和庫象徵性地鏈接到測試項目中。收集FlatCC後,您不需要CMAKE就可以構建自己的項目。
要創建另一個名為Foobar的測試項目,請調用scripts/setup.sh -s -x ../foobar 。這將避免從頭開始重建FlatCC項目。
注意:請參閱ChangElog。由於發現了API不一致,因此偶爾會有較小的破裂變化。除非明確說明,否則破壞更改不會影響編譯的運行時庫,而只會影響標頭文件。如果遇到麻煩,請確保flatcc工具與include/flatcc路徑相同。
該項目包括:
flatcc FlatBuffers架構編譯器,用於C和相應的庫libflatcc.a 。編譯器生成C標頭文件或二進制flatbuffers架構。libflatccrt.a用於構建和驗證C的Flatbuffers。生成的構建器標頭取決於此庫。它也可能對其他語言界面有用。該圖書館維護堆棧狀態,以使從解析器或類似產品構建緩衝區變得易於構建。flatcc/portable標頭庫,以及用於所有編譯器的小型助手,包括Endian處理和數字打印和解析。參見:
報告錯誤
Google FlatBuffers
建立說明
Quickstart
構建器接口參考
基準
flatcc編譯器被實現為獨立工具,而不是擴展Google flatc編譯器,以便具有純粹的Portable C庫的實現該模式編譯器,該架構編譯器的實現旨在長期運行過程中的濫用輸入而失敗。還認為,C版本可能有助於提供模式解析,以比C ++更容易找到與C更容易接口的語言接口。 Googles FPL實驗室的Flatbuffers團隊在提供反饋和回答許多問題以幫助確保最佳兼容性方面非常有幫助。請注意,名稱flatcc (FlatBuffers C編譯器)與Google flatc 。
JSON格式與Google flatc工具兼容。 flatc工具使用架構和緩衝區作為輸入將命令行的JSON轉換。 flatcc生成架構特定代碼以在運行時讀取和編寫JSON。儘管flatcc方法可能更快且更易於部署,但在使用JSON(例如編輯遊戲場景)手動工作時, flatc方法可能更方便。這兩個工具都有自己的位置。
注意:截至版本0.4.0,大型平台僅受支持。
它被認為是對Meson Build System的增加的支持,但是對此Via第56條的反饋會很好
如果可能的話,請提供一個簡短的可重複模式和源文件,並帶有主程序,返回1時返回1,成功的腳本為0和一個小的構建腳本。最好生成六個人並調用緩衝區驗證器,以確保輸入有效,並與debug庫flatccrt_d鏈接。
另請參閱調試緩衝區,然後readfile.h可用於讀取現有的緩衝區以進行驗證。
例子:
樣本/bugreport
折衷的fbs:
namespace Eclectic ;
enum Fruit : byte { Banana = -1 , Orange = 42 }
table FooBar {
meal : Fruit = Banana ;
density : long ( deprecated );
say : string ;
height : short ;
}
file_identifier "NOOB" ;
root_type FooBar ;Myissue.C:
/* Minimal test with all headers generated into a single file. */
#include "build/myissue_generated.h"
#include "flatcc/support/hexdump.h"
int main ( int argc , char * argv [])
{
int ret ;
void * buf ;
size_t size ;
flatcc_builder_t builder , * B ;
( void ) argc ;
( void ) argv ;
B = & builder ;
flatcc_builder_init ( B );
Eclectic_FooBar_start_as_root ( B );
Eclectic_FooBar_say_create_str ( B , "hello" );
Eclectic_FooBar_meal_add ( B , Eclectic_Fruit_Orange );
Eclectic_FooBar_height_add ( B , -8000 );
Eclectic_FooBar_end_as_root ( B );
buf = flatcc_builder_get_direct_buffer ( B , & size );
#if defined( PROVOKE_ERROR ) || 0
/* Provoke error for testing. */
(( char * ) buf )[ 0 ] = 42 ;
#endif
ret = Eclectic_FooBar_verify_as_root ( buf , size );
if ( ret ) {
hexdump ( "Eclectic.FooBar buffer for myissue" , buf , size , stdout );
printf ( "could not verify Electic.FooBar table, got %sn" , flatcc_verify_error_string ( ret ));
}
flatcc_builder_clear ( B );
return ret ;
}build.sh:
#! /bin/sh
cd $( dirname $0 )
FLATBUFFERS_DIR=../..
NAME=myissue
SCHEMA=eclectic.fbs
OUT=build
FLATCC_EXE= $FLATBUFFERS_DIR /bin/flatcc
FLATCC_INCLUDE= $FLATBUFFERS_DIR /include
FLATCC_LIB= $FLATBUFFERS_DIR /lib
mkdir -p $OUT
$FLATCC_EXE --outfile $OUT / ${NAME} _generated.h -a $SCHEMA || exit 1
cc -I $FLATCC_INCLUDE -g -o $OUT / $NAME $NAME .c -L $FLATCC_LIB -lflatccrt_d || exit 1
echo " running $OUT / $NAME "
if $OUT / $NAME ; then
echo " success "
else
echo " failed "
exit 1
fi 版本0.6.2(在開發中)主要是錯誤修復版本,有關詳細信息,請參閱ChangElog。已經確定了一個長期存在的錯誤,在對像在調用_create_as_root之前創建的對象將無法正確對齊,而現在也將緩衝區端填充到緩衝區內看到的最大對象。請注意,對於clang debug builds,添加了-fsanitize = undefined,這可能需要依賴的源代碼,也需要使用該標誌來避免缺少鏈接器符號。該功能可以在cmakelists.txt中禁用。
版本0.6.1主要包含錯誤修復和社區的大量貢獻,以處理平台邊緣案例。此外,Pendantic GCC警告是禁用的,而是依賴於Clang,因為GCC過於侵略性,因此斷裂經常建立並不利於便攜性。現有的C ++測試用例可確保C代碼還可以與常見的C ++編譯器一起使用,但是它可以破壞某些環境,因此現在有一個標誌可以禁用該測試而無需禁用所有測試。已經添加了對FlatBuffer格式中可選標量值的支持。在各種平台上,還可以提高支持記憶分配的支持。 <table>_identifier已在生成的代碼中對<table>_file_identifier偏愛,因為identifier很容易導致姓名衝突。現在,生成的代碼中的file_extension常數沒有前綴點(。)。
版本0.6.0引入了一個“主”屬性,該屬性與密鑰屬性一起使用,以選擇用於查找和排序的默認鍵。如果不存在主,則具有最低ID的鍵將變為主要。現在可以在主密鑰上遞歸對錶和向量進行排序。打破:以前第一個列出的不是最低的ID是主要鍵。還引入了結構字段中的固定長度標量陣列(不支持結構和枚舉元素)。結構支持固定長度數組字段,包括char數組。空結構從未完全工作,也不再支持,它們也不再受FlatC的支持。注意:Char數組當前不是Googles FlatC編譯器的一部分-INT8陣列可以使用。打破:不再支持空結構 - 它們在Googles FlatC編譯器中也無效。有關其他更改,請參見ChangElog。棄用: flatcc_accessors.h中的低級cast_to/from功能中刪除,以刪除read/write_from/to因為鑄件接口斷開某些罕見平台上的浮點轉換。這不應影響正常使用,但在此版本中仍然有效。
發行0.5.3插入各種錯誤修復程序(請參見ChangElog)和一個破壞但可能的影響較低的影響:破壞:BREAKING:BREAKING:0.5.3構建器的行為創建呼叫,因此在使用ID屬性時始終按字段ID訂購參數,例如MyGame_Example_Monster_create() monster_test.fbs (#81))。通過數字鍵字段對錶進行排序時修復了未定義的行為。
版本0.5.2向閱讀器方法引入可選的_get 。通過使用flatcc -g只有_get方法是有效的。這刪除了某些字段名稱的潛在名稱。 0.5.2還引入了期待已久的克隆操作,用於表和向量。添加了一個C ++煙節,以減少數字void指針分配錯誤,但不斷潛入。運行時庫現在需要額外的文件refmap.c 。
版本0.5.1修復了JSON打印機中的緩衝區超支,並改進了便攜式庫<Stdalign.h>與C ++和嵌入式newlib標準庫的兼容性。 JSON打印和解析已更加一致,可以幫助解析和打印表格以外的其他架構根,如Test_json.c中的測試驅動程序所示。 Monster_test.fbs文件已重新組織,以使Monster表與Googles FlatC版本更加一致,因此,較小的模式名稱空間不一致得以解決。對便攜式標頭的明確引用已移出生成的來源。外部“ C” C ++後衛在生成的標頭周圍增加了。 0.5.1還清理了低級聯合接口,因此{type,value}的術語在{type,embly}和{type,embers}上始終如一地使用。
be 。一旦項目變得穩定,就沒有計劃進行頻繁更新,但是社區的輸入將始終受到歡迎,並包含在相關的發行版中,尤其是在不同目標平台上測試。
此列表有些過時,添加了最新的編譯器版本,當CI平台不再支持時,刪除了一些舊版本,但很大程度上受支持的目標保持不變。 MSVC 2010將來可能會被棄用。
CI-More分支測試其他編譯器:
C11/C ++ 11是期望始終工作的參考。
在GCC-8+的情況下,不支持GCC --pedantic編譯器選項,因為它會迫使不可攜帶的代碼更改,並且由於每個新的GCC發布都會破壞代碼庫。
MSVC 2017並不總是經過測試,因為CI環境將不支持MSVC 2010。
C ++編譯器的較舊/非標準版本會引起問題,因為static_assert和alignas以奇怪的方式行事,在這種方式中,它們既不存在也不如預期的那樣完全工作。通常會有解決方法,但是使用-std=c++11或-std=c++14更可靠。
該庫中的庫不支持GCC C ++ PRE 4.7,因為便攜式庫在stdalign.h和stdint.h中的C ++限制不起作用。這可以解決,但不是優先事項。
隨著CI環境的更新,可能已經退休了一些先前的Testet編譯器版本。有關當前配置,請參見ci-more分支中的.travis.yml和appveyor.yml 。
怪物樣本與MSVC 2010不使用,因為它有意使用C99樣式代碼來更好地遵循C ++版本。
構建選項FLATCC_TEST可用於禁用所有可能使FlatCC編譯在原本有問題的平台上的測試。可以專門用於C ++測試(包括生成C代碼的簡單的C ++文件),可以禁用大量選項FLATCC_CXX_TEST 。
沒有理由不能支持其他或較舊的編譯器,但是它可能需要在構建配置中進行一些工作,並且可能需要更新到便攜式庫。以上只是已測試和配置的內容。
便攜性層具有一些對於諸如endian處理之類的功能,而其他功能則具有可选和缺失C11功能的兼容性。這應該支持大多數C編譯器,但依靠社區反饋來成熟。
可以通過使用-STD = C11並避免使用JSON(需要大量的數字解析支持)來大大減少運行時的必要大小,並刪除include/flatcc/reflection ,它可以支持二進制架構文件,並且可以從reflection/reflection.fbs / include/flatcc/support sworts and flatcc/spects and and and spects and Same和Same and Same和Same and Same and Same。確切的必需文件可能會隨著發布而變化,並且在編譯的代碼大小方面並不重要。
優先級是設計一個易於使用的C構建器界面,該界面相當快,適用於服務器和嵌入式設備,但是具有絕對性能的可用性 - 仍然以每秒毫秒為單位測量了小緩衝速度,並從粗略的估計中讀取10-100 millon緩衝液。讀取Flatbuffer的範圍不僅僅是構建它們的數量級。
對於具有1000個怪物,動態擴展的怪物名稱,怪物向量和庫存向量的100MB緩衝區,帶寬在2.2GHz Haswell Core core i7 cpu上達到約2.2GB/s和45ms/buffer。這包括讀取並驗證所有數據。僅讀取幾個關鍵字段將帶寬提高到2.7GB/s和37ms/op。對於10MB緩衝區,帶寬可能更高,但最終將被呼叫開銷擊中較小的緩衝區,因此我們以約150ns/op編碼小型緩衝區的速度下降到300MB/s。這些數字只是一個粗略的指南 - 它們顯然取決於硬件,編譯器和編碼數據。測量值不包括初始熱身步驟。
生成的JSON解析器的速度比直接在C或C ++中構建FlatBuffer的速度約4倍,或者大約是2200NS vs 600NS,用於700字節JSON消息。因此,JSON解析比在Google Flatbuffers基準頁面上報導的等效協議緩衝區大約要快兩個數量級。 LZ4壓縮將估計JSON解析的總體處理時間兩倍。 JSON打印比解析更快,但不是很明顯。 JSON壓縮到大約一半的大型緩衝液在壓縮的扁平按鈕上的大小,但在小型緩衝液上壓縮更糟(在根本不壓縮時更不用說)。
應當指出的是,FlatBuffer閱讀性能排除了JSON解析和協議緩衝區本質上包含的驗證。驗證尚未得到基準測試,但可能會增加少於50%的讀取開銷,除非僅讀取大型緩衝液的一小部分。
另請參閱基準。
客戶端C代碼幾乎可以避免任何形式的分配來構建緩衝區,因為建築商堆棧提供了可擴展的競技場,然後再進行對象 - 例如,附加字符串或向量零碎。當可以直接構造一個完整的對象時,例如在小Endian平台上的整數數組中的向量,大部分都會繞過堆棧。
讀者界面應該非常快,而明智的改進性能的空間也較小。它也比構建器簡單得多。
可用性還優先於最小的生成源代碼和編譯時間。它不應影響彙編的大小。
除了最限制的微控制器以外,所有的二進制輸出應相當小。一個33K怪物源測試文件(除了生成的標頭和構建器庫)導致不到50k優化的二進制可執行文件,包括printf語句和其他支持邏輯的開銷,或不包括構建器庫的30k對象文件。
僅讀取的二進製文件較小,但不一定比建築商考慮的工作要少得多:兼容性測試讀取預先生成的二進制monsterdata_test.golden Monster文件,並驗證所有內容是否如預期的。這將產生13K優化的二進制可執行文件或6K對象文件。此檢查的來源是5K不包括標頭文件。讀者不需要與庫鏈接。
JSON解析器與純flatbuffer使用量相比,將編譯的二進製文件膨脹,因為它們會嵌入解析器決策樹。 Monster.fbs的JSON解析器可以將100K +/-優化設置添加到可執行的二進制中。
生成的用於構建FlatBuffer的代碼以及用於解析和打印FlatBuffers的代碼,都需要訪問include/flatcc 。讀者不依賴任何庫,但是所有其他生成的文件都依賴於libflatccrt.a運行時庫。請注意,僅當需要flatcc編譯器本身作為庫時,才需要libflatcc.a 。
讀者和建築商依靠生成的普通讀取器和構建器標題文件。這些常見的文件使更改全局名稱空間並重新定義基本類型( uoffset_t等)成為可能。將來,這可能會進入庫代碼並將宏用於這些抽象,並最終為超出標準32位未簽名偏移量( uoffset_t )的類型提供了一組預定義的文件。運行時庫是特定於一組類型定義的。
有關詳細的使用指南,請參閱Monster_test.c和生成的文件。該項目中使用的怪物架構對原件進行了略微適應,以測試一些其他邊緣案例。
對於構建FlatBuffer,每個模式都會生成單獨的構建器標頭文件。它需要一個編譯器和小型運行時庫libflatccrt.a生成的flatbuffers_common_builder.h文件。正是由於這種要求,讀者和構建器生成的代碼被分開保留。典型用途可以在monster_test.c文件中看到。構建器允許在更新包含表的同時重複將內容推向向量或字符串,從而簡化了外部格式的解析。也可以在線構建嵌套緩衝液 - 起初這聽起來可能會過大,但是在將緩衝區的結合在網絡接口中並確保所有緩沖級別的對齊方式時很有用。
為了驗證flatbuffer,生成myschema_verifier.h 。這取決於運行時庫和讀取器標題。
JSON解析器和打印機每個架構文件生成一個文件,其中包括模式將有其自己的解析器和打印機,其中包括解析器和打印機將依靠,與建築商的工作方式相似。
低級別的注意:構建器在緩衝區的末端生成所有VTABLE,而不是在每個桌子前的臨時臨時生成,但否則可以對VTABLES進行相同的重複數據刪除。這使得可以將VTABLES聚集在熱緩存中,或確保在部分傳輸緩衝區時可用所有VTABLES。可以通過運行時標誌禁用此行為。
由於某些用例可能包含非常受限的嵌入式設備,因此可以使用分配器對象和緩衝區發射極對象自定義構建器庫。單獨的發射極可確保可以構建緩衝區,而無需單獨使用完整的緩衝區,如果需要的話。
在FlatCC_Builder.H和Flatcc_emitter.h中記錄了無類型的構建器庫,而生成的鍵入構建器API則記錄在構建器接口參考中。
有時,人們對生成代碼中使用的宏的密集性提出了一個問題。這些宏使很難理解哪些功能實際上可用。構建器接口參考試圖以一般方式記錄操作。為了獲取更詳細的信息,可以使用scripts/flatcc-doc.sh腳本提取生成的功能原型。
有些人還關注宏是“不安全”。與FlatCC一起使用時,宏不是不安全的,因為它們會生成靜態或靜態內聯函數。如果錯誤地使用與直接C代碼相同的擴展,這些將觸發編譯時間錯誤。
擴展將生成的輸出壓縮超過因子10,以確保源控制下的代碼不會爆炸,並可以以有意義的方式比較生成的代碼版本,並查看它是否與預期的架構匹配。宏對於通過便攜式標頭處理平台抽像也很重要。
儘管如此,儘管不直接由構建系統支持,但仍可以看到生成的輸出。例如, include/flatcc/reflection包含反射架構的預生成的標頭文件。要使用clang編譯器工具鏈查看擴展的輸出,請運行:
clang -E -DNDEBUG -I include
include/flatcc/reflection/reflection_reader.h |
clang-format
在不支持Clang的平台上可能還可以使用其他類似的命令。
請注意,編譯器將優化幾乎所有生成的代碼,並且僅使用最終用戶代碼實際引用的邏輯,因為功能是靜態的或靜態的。其餘部分通常會在應用程序代碼中有效內聯,從而導致相當小的二進制代碼大小。
更多細節可以在#88中找到
生成代碼的擴展可用於獲取特定對像類型的文檔。
以下腳本自動化了此過程:
scripts/flatcc-doc.sh <schema-file> <name-prefix> [<outdir>]
寫入函數原型<outdir>/<name-prefix>.doc 。
請注意,該腳本需要Clang編譯器和Clang-Format工具,但是該腳本也可能適用於其他工具鏈。
可以使用反射模式作為一個例子來說明腳本背後的原理,其中提取了對象表的文檔:
bin/flatcc reflection/reflection.fbs -a --json --stdout |
clang - -E -DNDEBUG -I include |
clang-format -style="WebKit" |
grep "^static.* reflection_Object_w*(" |
cut -f 1 -d '{' |
grep -v deprecated |
grep -v ");" |
sed 's/__tmp//g' |
sed 's/)/);/g'
Clang-Format的WebKit樣式確保參數和返回類型都放在同一行上。 GREP提取功能標頭,並切割帶有從同一行開始的函數機構。 SED Strips __tmp後綴來自用於避免宏名稱衝突的參數名稱。格雷普條);要刪除冗餘的正向聲明,然後添加;使每行成為有效的C原型。
不能保證以上內容始終可以隨著輸出而變化,但應該會有很長的路要走。
輸出的一小部分,如FlatCC-V0.5.2
static inline size_t reflection_Object_vec_len(reflection_Object_vec_t vec);
static inline reflection_Object_table_t reflection_Object_vec_at(reflection_Object_vec_t vec, size_t i);
static inline reflection_Object_table_t reflection_Object_as_root_with_identifier(const void* buffer, const char* fid);
static inline reflection_Object_table_t reflection_Object_as_root_with_type_hash(const void* buffer, flatbuffers_thash_t thash);
static inline reflection_Object_table_t reflection_Object_as_root(const void* buffer);
static inline reflection_Object_table_t reflection_Object_as_typed_root(const void* buffer);
static inline flatbuffers_string_t reflection_Object_name_get(reflection_Object_table_t t);
static inline flatbuffers_string_t reflection_Object_name(reflection_Object_table_t t);
static inline int reflection_Object_name_is_present(reflection_Object_table_t t);
static inline size_t reflection_Object_vec_scan_by_name(reflection_Object_vec_t vec, const char* s);
static inline size_t reflection_Object_vec_scan_n_by_name(reflection_Object_vec_t vec, const char* s, int n);
...
使用反射和怪物模式在以下腳本中提供了示例:
scripts/reflection-doc-example.sh
scripts/monster-doc-example.sh
Monster Doc示例本質上調用:
scripts/flatcc-doc.sh samples/monster/monster.fbs MyGame_Sample_Monster_
導致文件MyGame_Sample_Monster_.doc :
static inline size_t MyGame_Sample_Monster_vec_len(MyGame_Sample_Monster_vec_t vec);
static inline MyGame_Sample_Monster_table_t MyGame_Sample_Monster_vec_at(MyGame_Sample_Monster_vec_t vec, size_t i);
static inline MyGame_Sample_Monster_table_t MyGame_Sample_Monster_as_root_with_identifier(const void* buffer, const char* fid);
static inline MyGame_Sample_Monster_table_t MyGame_Sample_Monster_as_root_with_type_hash(const void* buffer, flatbuffers_thash_t thash);
static inline MyGame_Sample_Monster_table_t MyGame_Sample_Monster_as_root(const void* buffer);
static inline MyGame_Sample_Monster_table_t MyGame_Sample_Monster_as_typed_root(const void* buffer);
static inline MyGame_Sample_Vec3_struct_t MyGame_Sample_Monster_pos_get(MyGame_Sample_Monster_table_t t);
static inline MyGame_Sample_Vec3_struct_t MyGame_Sample_Monster_pos(MyGame_Sample_Monster_table_t t);
static inline int MyGame_Sample_Monster_pos_is_present(MyGame_Sample_Monster_table_t t);
static inline int16_t MyGame_Sample_Monster_mana_get(MyGame_Sample_Monster_table_t t);
static inline int16_t MyGame_Sample_Monster_mana(MyGame_Sample_Monster_table_t t);
static inline const int16_t* MyGame_Sample_Monster_mana_get_ptr(MyGame_Sample_Monster_table_t t);
static inline int MyGame_Sample_Monster_mana_is_present(MyGame_Sample_Monster_table_t t);
static inline size_t MyGame_Sample_Monster_vec_scan_by_mana(MyGame_Sample_Monster_vec_t vec, int16_t key);
static inline size_t MyGame_Sample_Monster_vec_scan_ex_by_mana(MyGame_Sample_Monster_vec_t vec, size_t begin, size_t end, int16_t key);
...
也可以提取FlatBuffer本地類型,例如字符串操作:
scripts/flatcc-doc.sh samples/monster/monster.fbs flatbuffers_string_
導致flatbuffers_string_.doc :
static inline size_t flatbuffers_string_len(flatbuffers_string_t s);
static inline size_t flatbuffers_string_vec_len(flatbuffers_string_vec_t vec);
static inline flatbuffers_string_t flatbuffers_string_vec_at(flatbuffers_string_vec_t vec, size_t i);
static inline flatbuffers_string_t flatbuffers_string_cast_from_generic(const flatbuffers_generic_t p);
static inline flatbuffers_string_t flatbuffers_string_cast_from_union(const flatbuffers_union_t u);
static inline size_t flatbuffers_string_vec_find(flatbuffers_string_vec_t vec, const char* s);
static inline size_t flatbuffers_string_vec_find_n(flatbuffers_string_vec_t vec, const char* s, size_t n);
static inline size_t flatbuffers_string_vec_scan(flatbuffers_string_vec_t vec, const char* s);
static inline size_t flatbuffers_string_vec_scan_n(flatbuffers_string_vec_t vec, const char* s, size_t n);
static inline size_t flatbuffers_string_vec_scan_ex(flatbuffers_string_vec_t vec, size_t begin, size_t end, const char* s);
...
有關詳細信息,請參閱flatcc -h 。
此處列出的在線版本:Flatcc -help.md,但請使用flatcc -h進行最新參考。
編譯器可以生成一個單個標頭文件,也可以為所有包含的架構和一個通用文件以及不支持閱讀(默認)和寫入(-W)FlatBuffers的標題。最簡單的選項是對所有人使用(-a),並包括myschema_builder.h文件。
(-a)或(-v)還會生成一個驗證者文件。
確保在c編譯器中可見包含include夾下的flatcc在編譯FlatBuffer構建器時包括路徑。
flatcc (-i)Include路徑將假設所有具有相同基本名稱(案例insentive)的架構文件是相同的,並且僅包括第一個。所有生成的文件均使用輸入Basename,並將降落在工作目錄或(-o)設置的路徑中。
可以使用( - stdout)生成文件以生成stdout。 C標題將被訂購和連接,但與單獨的文件輸出相同。每個包含的語句都受到保護,因此這不會導致丟失的包括文件。
生成的代碼,尤其是所有與-stdout結合在一起的代碼可能會顯示大,但是實際上實際使用的零件才能在最終的可執行文件或對象文件中佔用空間。現代編譯器內聯,僅包含靜態鏈接的構建器庫的必要部分。
如果僅需要一個,則可以使用-json flag或-json-printer或json-parser生成JSON打印機和解析器。有一些運行時庫編譯時標誌可以優化打印符號枚舉,但是在運行時也可以禁用這些枚舉。
確保與libflatccrt (用於運行時的RT)而不是libflatcc (架構編譯器)鏈接,否則構建器將不可用。另外,請確保將flatcc項目root的“包含”在包含路徑中。
默認情況下,FlatCC將在閱讀或驗證緩衝區時期望緩衝區中的file_identifier 。
緩衝區可以在偏移4時具有意外的4字節標識符,或者可能不存在標識符。
並非所有語言界面都支持緩衝區中的文件標識符,如果這樣做,它們可能不會在較舊的版本中使用。用戶報告了Python和LUA接口的問題,但這很容易解決。
檢查驗證者的返回值:
int ret;
char *s;
ret = MyTable_verify_as_root(buf, size);
if (ret) {
s = flatcc_verify_error_string(ret);
printf("buffer failed: %sn", s);
}
要驗證沒有標識符的緩衝區,或者忽略了其他標識符,請使用null標識符使用驗證程序的_with_identifier版本:
char *identifier = 0;
MyTable_verify_as_root_with_identifier(buf, size, identifier);
閱讀緩衝區使用:
MyTable_as_root_with_identifier(buf, 0);
並在沒有標識符的情況下構建緩衝區:
MyTable_start_as_root_with_identifier(builder, 0);
...
MyTable_end_as_root_with_identifier(builder, 0);
其他幾個as_root調用具有as_root_with_identifier版本,包括JSON打印。
構建flatcc tool後,二進製文件位於flatcc源樹下方的bin和lib目錄中。
您可以直接跳到遵循Google flatbuffers教程的怪物示例,也可以沿著下面的Quickstart指南閱讀。如果您遵循Monster教程,則可能需要克隆並構建FlatCC並將源複製到單獨的項目目錄,如下所示:
git clone https://github.com/dvidelabs/flatcc.git
flatcc/scripts/setup.sh -a mymonster
cd mymonster
scripts/build.sh
build/mymonster
scripts/setup.sh將最小鏈接庫和工具鏈接到mymonster自定義目錄。使用(-a)還添加了一個簡單的scripts/setup.sh -h腳本,複製示例和更新.gitignore設置也可以構建FlatCC,但是您仍然必須確保為系統配置構建環境。
要編寫您自己的模式文件,請按照編寫模式文件的“ flatbuffers”項目文檔。
在研究怪物樣本和下面的快速啟動後,構建器界面參考可能很有用。
在尋找高級示例時,例如對向量進行排序和通過密鑰查找元素,您應該在test/monster_test項目中找到這些元素。
以下快速入門指南是對test/monster_test項目的廣泛簡化 - 請注意,該模式與教程略有不同。專注於C特定框架,而不是一般的FlatBuffers概念。
您仍然可以使用設置工具來創建一個空項目並遵循,但是在下面的文本中沒有任何假設。
在這裡,我們提供了一個僅閱讀訪問monster flatbuffer的快速示例 - 它是monster_test.c文件的改編提取物。
首先,我們使用Common(-C)支持標頭編譯架構,並添加遞歸,因為monster_test.fbs包含其他文件。
flatcc -cr --reader test/monster_test/monster_test.fbs
為簡單起見,我們假設您在項目根文件夾中構建示例項目,但是在實踐中,您需要更改一些路徑,例如:
mkdir -p build/example
flatcc -cr --reader -o build/example test/monster_test/monster_test.fbs
cd build/example
我們得到:
flatbuffers_common_reader.h
include_test1_reader.h
include_test2_reader.h
monster_test_reader.h
(還有更簡單的samples/monster/monster.fbs ,但是您將不會收到架構文件)。
名稱空間可以很長,因此我們可以選擇使用宏來管理它。
#include "monster_test_reader.h"
#undef ns
#define ns(x) FLATBUFFERS_WRAP_NAMESPACE(MyGame_Example, x)
int verify_monster(void *buffer)
{
ns(Monster_table_t) monster;
/* This is a read-only reference to a flatbuffer encoded struct. */
ns(Vec3_struct_t) vec;
flatbuffers_string_t name;
size_t offset;
if (!(monster = ns(Monster_as_root(buffer)))) {
printf("Monster not availablen");
return -1;
}
if (ns(Monster_hp(monster)) != 80) {
printf("Health points are not as expectedn");
return -1;
}
if (!(vec = ns(Monster_pos(monster)))) {
printf("Position is absentn");
return -1;
}
/* -3.2f is actually -3.20000005 and not -3.2 due to representation loss. */
if (ns(Vec3_z(vec)) != -3.2f) {
printf("Position failing on z coordinaten");
return -1;
}
/* Verify force_align relative to buffer start. */
offset = (char *)vec - (char *)buffer;
if (offset & 15) {
printf("Force align of Vec3 struct not correctn");
return -1;
}
/*
* If we retrieved the buffer using `flatcc_builder_finalize_aligned_buffer` or
* `flatcc_builder_get_direct_buffer` the struct should also
* be aligned without subtracting the buffer.
*/
if (vec & 15) {
printf("warning: buffer not aligned in memoryn");
}
/* ... */
return 0;
}
/* main() {...} */
假設我們上面的文件是monster_example.c c以下是編譯項目僅閱讀的幾種方法 - 稍後顯示使用運行時庫的編譯。
cc -I include monster_example.c -o monster_example
cc -std=c11 -I include monster_example.c -o monster_example
cc -D FLATCC_PORTABLE -I include monster_example.c -o monster_example
包括路徑或源路徑可能不同。始終使用include/flatcc/portable中的某些文件,但是-D FLATCC_PORTABLE標誌包含其他文件,以支持缺少C11功能的編譯器。
注意:在某些clang/GCC平台上,如果鏈接器無法找到posix_memalign ,則可能有必要使用-std = gnu99或-std = gnu11,另請參見paligned_alloc.h中的註釋。
在這裡,我們提供了一個非常有限的示例,說明如何構建緩衝區 - 只有幾個字段已更新。請參閱Monster_test.c和DOC目錄以獲取更多信息。
首先,我們必須生成文件:
flatcc -a monster_test.fbs
這會產生:
flatbuffers_common_reader.h
flatbuffers_common_builder.h
include_test1_reader.h
include_test1_builder.h
include_test1_verifier.h
include_test2_reader.h
include_test2_builder.h
include_test2_verifier.h
monster_test_reader.h
monster_test_builder.h
monster_test_verifier.h
注意:除非我們只打算讀取緩衝區,否則我們實際上不會進行前面顯示的可讀生成 - 建築商的生成也總是會生成讀取acpes。
通過包括"monster_test_builder.h"所有其他文件都會自動包含。 C編譯器需要-I include指令以訪問flatcc/flatcc_builder.h , flatcc/flatcc_verifier.h和其他文件,並根據細節的不同文件,假設項目根是當前目錄。
不需要驗證符,只是因為我們懶惰地選擇了-a選項,因此不需要驗證器。
必須先初始化構建器才能設置我們有效構建緩衝區所需的運行時環境 - 構建器依靠發射器對象來構建實際的緩衝區 - 在這裡,我們隱含地使用默認值。一旦有了這一點,我們就可以將構建器視為手柄,並專注於生成的API,直到最終確定緩衝區(即訪問結果)。對於非平凡用途,建議使用自定義的發射器,例如,在它們完成後立即在網絡上發布頁面,而不是使用flatcc_builder_finalize_buffer將所有頁面合併到單個緩衝區中,或者簡單的flatcc_builder_get_direct_buffer如果返回null(如果buffer均太大)。另請參見flatcc_builder.h和flatcc_emitter.h中的文檔註釋。另請參見builder.h中的flatc_builder_finalize_aligned_buffer ,當malloc對齊的緩衝區不足時,構建器接口參考。
#include "monster_test_builder.h"
/* See [monster_test.c] for more advanced examples. */
void build_monster(flatcc_builder_t *B)
{
ns(Vec3_t *vec);
/* Here we use a table, but structs can also be roots. */
ns(Monster_start_as_root(B));
ns(Monster_hp_add(B, 80));
/* The vec struct is zero-initalized. */
vec = ns(Monster_pos_start(B));
/* Native endian. */
vec->x = 1, vec->y = 2, vec->z = -3.2f;
/* _end call converts to protocol endian format - for LE it is a nop. */
ns(Monster_pos_end(B));
/* Name is required, or we get an assertion in debug builds. */
ns(Monster_name_create_str(B, "MyMonster"));
ns(Monster_end_as_root(B));
}
#include "flatcc/support/hexdump.h"
int main(int argc, char *argv[])
{
flatcc_builder_t builder;
void *buffer;
size_t size;
flatcc_builder_init(&builder);
build_monster(&builder);
/* We could also use `flatcc_builder_finalize_buffer` and free the buffer later. */
buffer = flatcc_builder_get_direct_buffer(&builder, &size);
assert(buffer);
verify_monster(buffer);
/* Visualize what we got ... */
hexdump("monster example", buffer, size, stdout);
/*
* Here we can call `flatcc_builder_reset(&builder) if
* we wish to build more buffers before deallocating
* internal memory with `flatcc_builder_clear`.
*/
flatcc_builder_clear(&builder);
return 0;
}
編譯示例項目:
cc -std=c11 -I include monster_example.c lib/libflatccrt.a -o monster_example
請注意,構建緩衝區需要運行時庫,但不需要閱讀它們。如果為給定目標分配運行時庫是不具體的,則可以使用源文件。每個功能都有其自己的源文件,因此構建緩衝區並不需要所有運行時文件:
cc -std=c11 -I include monster_example.c
src/runtime/emitter.c src/runtime/builder.c
-o monster_example
其他功能,例如驗證者和JSON打印機和解析器,每個功能都需要在SRC/運行時需要一個不同的文件。從文件名中可以明顯看出哪個文件,除了JSON解析還需要構建器和發射器源文件。
可以驗證緩衝區,以確保其不包含在給定緩衝區大小之外的任何範圍,即根據Flatbuffer原理對所有數據結構進行對齊,即字符串為零終止,並且存在所需的字段。
在上面的構建器示例中,我們可以將驗證者應用於輸出:
#include "monster_test_builder.h"
#include "monster_test_verifier.h"
int ret;
...
... finalize
if ((ret = ns(Monster_verify_as_root_with_identifier(buffer, size,
"MONS")))) {
printf("Monster buffer is invalid: %sn",
flatcc_verify_error_string(ret));
}
readFile.h實用程序也可能有助於閱讀現有的緩衝區進行驗證。
FlatBuffers可以選擇遺漏標識符,在這裡“ Mons”。使用空指針作為標識符參數來忽略任何現有標識符並允許丟失的標識符。
嵌套的FlatBuffer始終通過空標識符進行驗證,但是稍後可以在訪問緩衝區時檢查。
驗證者未驗證兩個數據架構不是重疊的。有時,這確實是有效的,例如DAG(有向無環圖),例如,兩個字符串引用是指緩衝區中的同一字符串。在其他情況下,攻擊者可能會惡意構建重疊的數據架構,以便就地更新可能導致隨後的無效緩衝區。因此,如果不先將其重寫為新的緩衝區,則絕對不應將不受信任的緩衝區更新。
CMAKE構建系統具有構建選項,可以在驗證者中啟用斷言。這將打破調試構建,通常不是所需的內容,但是在調試為什麼緩衝區無效時,這可能非常有用。也可以啟用跡線,以便可以報告表偏移和字段ID。
另請參閱include/flatcc/flatcc_verifier.h 。
當驗證直接從構建器返回的緩衝區時,可能有必要使用flatcc_builder_finalize_aligned_buffer來確保正確對齊並使用aligned_free來釋放緩衝區(或按照v0.5.0,也是flatcc_builder_aligned_free ,請參見buililder buililder interface confife。緩衝區也可以通過MMAP複製到對齊的內存中,也可以使用便攜式圖層paligned_alloc.h功能,該功能在包括生成的標頭時可用。 test/flatc_compat/flatc_compat.c是如何完成此操作的一個示例。對於大多數用例,標準分配就足夠了,但是例如,僅在8字節邊界上分配標準的32位窗口,並且可以破壞怪物模式,因為它具有16個字節的對齊字段。
注意:截至2024年8月,已經發現,即使元素需要更大的對齊方式,c ++作者代碼也只能將空向量與大小字段對齊,即使需要更高的對齊方式(需要double類型)8。這會導致FlatCC驗證器(正確)拒絕這些矢量,因為它會導致對某些架構的無效c pointer類型。但是,由於這已經有效了10多年,因此共識是要使驗證者容忍這種行為,即使C ++最終將解決此問題。 FlatCC驗證器已更新以默認使用可選的編譯時標誌來接受此類緩衝區,以執行嚴格的行為( FLATCC_ENFORCE_ALIGNED_EMPTY_VECTORS )。原則上,未對準的矢量可能會導致在壓縮優化的C編譯器中不確定的行為。截至目前,在公共平台上閱讀此類緩衝區似乎是安全的,最好避免額外的運行時閱讀器開銷來處理此操作。有關更多信息,請參見Flatcc#287,Google FlatBuffers#8374,FlatCC#289。
如果不幸的話,可能會使讀取登錄方法與其他生成的方法和打字名稱衝突。通常,模式的小變化將解決此問題。
從flatcc 0.5.2開始,讀取ACCOR是帶有和不帶有_get後綴的,因此也可以使用Monster_pos_get(monster)而不是Monster_pos(monster) 。使用選項-g調用FlatCC時,只有_get後綴才能生成讀取配件。這避免了潛在的名稱衝突。衝突的一個示例是當還有pos字段時,例如pos_add字段名稱,因為構建器接口會生成add後綴。使用-g選項可以避免此問題,但是最好選擇其他名稱,例如added_pos時可以修改架構時。
-g選項僅更改flatbuffers_common_reader.h文件的內容,因此,如果不混合,從技術上講可以使用此文件的不同版本。
如果外部代碼生成器取決於FlatCC輸出,則應使用_get後綴,因為它可以使用和不使用-G選項,但僅在0.5.2或更高版本開始時使用。對於人類可讀的代碼,在沒有_get後綴的情況下堅持使用Orignal命名約定很容易。
即使在上述方面,仍然有可能與聯合類型字段發生衝突。如果一個聯合字段命名為foo ,則會自動一個額外的字段 - 此字段命名為foo_type ,毫不奇怪地將聯合類型固定。
名稱空間也可能導致衝突。如果模式具有名稱為字段名稱Hello的命名空間foo.bar和表格,則讀取訪問者將命名: Foo_Bar_MyTable_hello_get 。也可以使用一個名為Bar_MyTable的表,因為_在flatbuffers架構名稱中允許_,但是在這種情況下,我們在生成的C代碼中有名稱衝突。 FlatCC並不試圖避免這種衝突,因此這種模式被認為是無效的。
值得注意的是,有幾個用戶與名為“標識符”的表或結構字段遇到了衝突,因為<table-name>_identifier被定義為使用該表(或結構)作為root的緩衝區時使用的文件標識符。截至0.6.1,該名稱為<table-name>_file_identifier ,以減少衝突的風險。舊形式已被棄用,但仍會針對沒有名為“標識符”字段的表格生成,以使兼容性。大多數情況下,此宏用於更高級別的功能,例如mytable_create_as_root ,它需要知道要使用的標識符。
在閱讀flatbuffer並不能提供預期的結果時,第一道防線是確保正在測試的代碼與flatccrt_d鏈接(運行時庫的調試構建)。如果對建設者的電話不正確平衡或未設置需要的字段,這將提出主張。
要進一步研究緩衝區,請調用緩衝區驗證器,看看緩衝區相對於預期的緩衝區類型是否有效。
當未在緩衝區中設置相應的字段時,字符串和桌子將作為空指針返回。用戶代碼應該對此進行測試,但是在架構中臨時或永久設置required屬性也可能有所幫助。然後,構建器將在固定緩衝區時檢測到缺失的字段,而驗證者可以在現有的緩衝區中檢測到它們的缺失。
如果驗證者拒絕緩衝區,則可以打印錯誤(請參閱驗證緩衝區),但不會確切說明問題的位置。再進一步,可以做出驗證者來斷言遇到問題的位置,以便可以分析緩衝區的內容。啟用了:
-DFLATCC_DEBUG_VERIFY=1
請注意,這將打破預計驗證驗證失敗的測試用例。
要轉儲有效緩衝區的詳細內容,或在失敗點之前的有效內容,請使用:
-DFLATCC_TRACE_VERIFY=1
這兩個選項都可以設置為CMAKE選項,也可以在FlatCC_RTConfig.h文件中設置。
在報告錯誤時,上述輸出也可能證明是有幫助的。
JSON解析器和打印機也可用於創建和顯示緩衝區。解析器將正確使用構建器API,或在所需字段上簽發語法錯誤或錯誤。這可以排除正確使用API的一些不確定性。 test_json.c文件和test_json_parser.c具有可以適應自定義測試的測試功能。
對於高級調試,hexdump.h文件可用於丟棄緩衝區內容。它用於test_json.c,也用於monster_test.c。另請參見FlatBuffers二進制格式。
截至2022年4月,Googles FlatC工具已實施了--annotate功能。這提供了帶有二進制緩衝區和模式的註釋的十六進制轉儲。輸出可用於故障排除,排除或確認手頭緩衝區中的可疑編碼錯誤。 Flatbuffers二進制格式文檔中的折衷示例包含一個手寫的註釋示例,該示例啟發了--annotate功能,但並不是完全相同的輸出格式。還要注意, flatc生成的緩衝區傾向於在表格之前具有VTABLES,而FlatCC通常在緩衝區末端包裝所有VTABLE,以提高填充和緩存效率。
另請參閱flatc- andotate。
注意:對文本編輯器有實驗支持,該文本編輯器支持Clangd語言服務器或類似的實驗。您可以編輯CMakeList.txt以生成構建build/Debug/compile_comands.json ,至少在使用clang作為編譯器時,然後從root複製或symlink。或提出更好的建議。在項目root中,有compile_flags.txt和compile_commands.json的.gitignore條目。
有兩種方法可以識別FlatBuffer的內容。首先是使用架構中定義的文件標識符。第二個是使用type identifiers ,這些標識符是根據每個表名的名稱(以其名稱空間(如果有)為前綴)計算的。無論哪種情況,標識符都在二進制flatbuffers中存儲在偏移4中。類型標識符不要與工會類型混淆。
FlatBuffers架構語言具有可選的file_identifier聲明,該聲明接受4個字符ASCII字符串。它的目的是人類可讀。當不存在時,緩衝區可能會變短4個字節(取決於填充)。
file_identifier旨在匹配root_type架構聲明,但這並不考慮在其他類型上創建Flatbuffers很方便。 flatcc對root_type沒有特殊的目的地,而Google flatc json parser使用它來確定JSON root對像類型。
結果,文件標識符是模糊的。包含的模式可能具有單獨的file_identifier聲明。為了至少確保每種類型都與其自己的schemas file_identifier關聯,為每種類型定義了一個符號。如果模式具有這樣的標識符,則將定義為空標識符。
生成的代碼定義給定表的標識符:
#ifndef MyGame_Example_Monster_file_identifier
#define MyGame_Example_Monster_file_identifier "MONS"
#endif
現在,用戶可以覆蓋給定類型的標識符,例如:
#define MyGame_Example_Vec3_file_identifier "VEC3"
#include "monster_test_builder.h"
...
MyGame_Example_Vec3_create_as_root(B, ...);
create_as_root方法將標識符用於所討論的類型,其他_as_root方法也是如此。
file_extension以類似的方式處理:
#ifndef MyGame_Example_Monster_file_extension
#define MyGame_Example_Monster_file_extension "mon"
#endif
為了更好地處理文件標識符的歧義,已將類型標識符作為替代性4個字節緩衝區標識符引入。哈希在FNV-1A上進行標準化,以實現互操作性。
類型標識符使用類型的哈希,將完全合格的類型名稱映射到4個字節哈希中。類型哈希是一個32位本地值,類型標識符是一個4個字符的小endian編碼相同值的字符串。
在此示例中,類型哈希是從字符串“ mygame.example.monster”派生的,並且對於支持類型哈希的所有Flatbuffer代碼生成器而言,它們都是相同的。
值0用於指示一個人不在乎緩衝區中的標識符。
...
MyGame_Example_Monster_create_as_typed_root(B, ...);
buffer = flatcc_builder_get_direct_buffer(B);
MyGame_Example_Monster_verify_as_typed_root(buffer, size);
// read back
monster = MyGame_Example_Monster_as_typed_root(buffer);
switch (flatbuffers_get_type_hash(buffer)) {
case MyGame_Example_Monster_type_hash:
...
}
...
if (flatbuffers_get_type_hash(buffer) ==
flatbuffers_type_hash_from_name("Some.Old.Buffer")) {
printf("Buffer is the old version, not supported.n");
}
可以自然擴展現有的API自然可以使用。有關更多信息,請參見Monster_test.c。
類型標識符的定義如下:
#define MyGame_Example_Monster_type_hash ((flatbuffers_thash_t)0x330ef481)
#define MyGame_Example_Monster_type_identifier "x81xf4x0ex33"
可以在任何原始4個字符文件標識符的地方使用type_identifier ,但是緩衝區必須選擇要使用的系統(如果有)。這不會影響file_extension 。
注意:在生成的API中預期標識符字符串時,生成的_type_identifier字符串通常不應使用,因為它可能包含在比較之前第一個NULL之後將其填充為零的null字符。使用採用類型哈希的API調用。 type_identifier可以在低級別的flatcc_builder.h調用中使用,因為它將標識符處理為固定字節陣列和句柄類型的哈希和字符串相同。
注意:可以編譯FlatCC運行時,以大型endian格式編碼緩衝區,而不是標準的Little Endian格式,無論主機平台的endianness如何。如果完成此操作,則不管選擇哪種標識符方法,緩衝區中的標識字段總是被換成字節。 API調用使此透明,因此“ Mons”將被存儲為“ SNOM”,但仍應在API調用中驗證為“ Mons”。這種保護措施防止混合少量和大型緩衝區。同樣,始終以本機(主機)endian格式測試類型哈希。
flatcc/flatcc_identifier.h文件包含使用的FNV-1A哈希的實現。選擇哈希是為了簡單,可用性和碰撞阻力。為了更好地分配,還提供了分散功能,主要是為了阻止替代哈希在傳輸中的使用,因為哈希類型通常足夠好。
注意:類型的哈希值可能會發生碰撞,因為哈希只有4個字節。
JSON支持文件由flatcc --json生成。
本節不是關於JSON打印和解析的教程,它僅涵蓋了一些不太明顯的方面。快速入門的最佳來源是測試文件:
test/json_test/json_test.c
有關詳細的用法,請參考:
test/json_test/test_json_printer.c
test/json_test/test_json_parser.c
test/json_test/json_test.c
test/benchmark/benchflatccjson
另請參見Google FlatBuffers架構文檔中的JSON解析部分。
通過使用FlatBuffer模式,可以生成模式特定的JSON打印機和解析器。與Google flatc工具相比,這是更好和更差的,該工具將二進制模式作為輸入並處理JSON輸入和輸出。在這裡,解析器和打印機僅依賴flatcc運行時庫,速度更快(可能是顯著),但是當要支持新的JSON格式時需要重新編譯 - 這聽起來並不像聽起來那樣糟糕 - 例如,很難創建Docker容器來在Web服務器上下文中處理特定的架構。
解析器始終將文本緩衝區作為輸入,並根據構建器對象的初始化方式產生輸出。打印機具有不同的初始功能:一個用於打印到文件指針,包括stdout,一個用於打印到固定長度的外部緩衝區,另一個用於打印到動態增長的緩衝區。動態緩衝區可以通過重置功能在打印之間重複使用。有關詳細信息,請參見flatcc_json_parser.h 。
解析器將接受未引用的名稱(不是字符串)和尾隨逗號,即非圖案JSON,並且還允許在字符串中使用HEX x03 。必須通過編譯時標誌啟用嚴格模式。此外,解析器模式特定的符號枚舉值,在預期數值的情況下,可以選擇不可用:
color: Green
color: Color.Green
color: MyGame.Example.Color.Green
color: 2
不必引用符號值(除非運行時或編譯時間配置要求),但可以是在數值值時無法引用。如果沒有提供命名空間,例如color: Green ,則該符號必須與接收枚舉類型匹配。任何標量值都可以在相對名稱空間(例如hp: Color.Green )中接收一個符號值,或者像hp: MyGame.Example.Color.Green這樣的絕對名稱空間,而不是hp: Green (因為怪物示例示意圖中的hp )不是具有Green值的枚舉類型)。名稱空間相對於接收對象的命名空間。
也可以具有多個值,但是這些值總是必須引用,才能與Google flatC工具兼容用於FlatBuffers 1.1的FlatC工具:
color: "Green Red"
可以在編譯時啟用未引用color: Green Red多價枚舉
這些值值表示的表達式最初是針對具有定義的位標誌屬性(確實具有顏色)的枚舉,但這很難處理,因此可以以適當的或沒有名稱空間的情況為單位列出任何同義值。由於這進一步引起了簽名符號的問題,因此確切的定義是所有符號首先脅迫到目標類型(或失敗),然後將其添加到目標類型中,如果不是第一個,則該符號會導致:
color: "Green Blue Red Blue"
color: 19
因為綠色是2,所以紅色為1,藍色為8,重複。
注意:重複值應視為實現依賴性,因為不能保證所有Flatbuffer JSON解析器都將處理相同的功能。也許將來可能會改變此實現,例如使用位或所有成員和目標均為位標誌類型時。
指定一個空集不有效:
color: ""
因為它可能被理解為0或默認值,並且沒有很好地說明。
默認情況下,打印機將沒有任何空間和所有引用的所有空間。使用非圖案格式選項(請參閱標題和測試示例)來產生漂亮的打印。使用noenum選項禁用符號枚舉值。
只有枚舉才能打印符號值,根本沒有任何解析符號值的歷史記錄。此外,僅當存儲的值映射到一個值,或者在位列表的情況下,才能打印符號值。對於Exmaple,如果解析color: Green Red將以"color":"Red Green"打印,而color: Green Blue Red Blue將以顏色為color:19 。
打印機和解析器都限於大約100個表嵌套水平和另外100個嵌套結構深度。這可以通過配置標誌更改,但由於操作是遞歸下降,因此必須適合運行時堆棧。超過限制將導致錯誤。
數值被強制為接收類型。如果分配不適合目標,而浮點值可能會靜靜地放鬆精度,則整數類型將失敗。整數類型永遠不會接受浮點值。字符串只接受字符串。
嵌套的FlatBuffer可以通過字節大小的整數或目標類型的表格或結構來嵌套。有關詳細信息,請參見測試用例。
默認情況下,解析器會在未知字段上失敗,但是也可以使用運行時選項靜靜地跳過。
工會很難解析。一個聯合是兩個JSON字段:一張表像往常一樣,一個枚舉表示具有_type後綴相同名稱的類型,並接受數字或符號類型代碼:
{
name: "Container Monster",
test_type: Monster,
test: { name: "Contained Monster" }
}
基於模式在monster_test.fbs中定義。
由於其他JSON處理器可能會對字段進行排序,因此可以在測試字段之後接收類型字段。解析器不存儲臨時數據架構。它直接構建一個flatbuffer。當類型遲到時,這是不可能的。通過在第一次通過時以跳過的字段來解析該字段,然後在已知類型後進行鍵入的背面跟踪第二次(只有兩次桌子解析,但對於嵌套工會,這仍然可以擴展)。不用說這會減慢解析。僅提供表字段或單獨類型字段是一個錯誤,除非類型NONE或0 ,在這種情況下,則不允許存在表。
從v0.5.0開始支持聯合向量。聯合向量表示為兩個向量,一個是表格的表格,一個具有類型的向量,類似於普通的工會。將類型向量放置在更有效的效率上,因為它避免了回溯。由於類型無的聯合在與工會的向量打交道時不能通過缺少表字段來表示,因此,如果該表中的類型在相應的類型向量中不存在,則表必須具有值null 。在其他情況下,應不存在表,而不是零。
這是JSON包含Monster root表的示例,其中包含一個名為manyany的聯合向量字段,這是Monster_test.fbs schema中Any工會的矢量:FBS架構:
{
"name": "Monster",
"manyany_type": [ "Monster", "NONE" ],
"manyany": [{"name": "Joe"}, null]
}
從v0.5.0開始,可以用RFC 4648中記錄的base64編碼字符串或base64url編碼字符串編碼和解碼類型[uint8] (aka [ubyte] )的向量。
將支持限制為[uint8]避免將二進制數據引入字符串,也避免處理其他類型的二進制數據的符號和endian編碼。此外,大於8位的值編碼的數組編碼不一定比base64效率低。
base64填充始終打印,解析時是可選的。空格,線路斷開,JSON字符串逃脫字符''或Base64(url)字母中未在base 64(URL)中的任何其他字符被拒絕為解析錯誤。
模式必須將屬性(base64)或(base64url)添加到持有向量的字段中,例如:
table Monster {
name: string;
sprite: [uint8] (base64);
token: [uint8] (base64url);
}
如果需要將更複雜的數據編碼為base64,例如結構的向量,則可以通過嵌套的flatbuffer進行[uint8]類型。
請注意,在某些用例中,可能希望將二進制數據讀取為基礎64的內存與8位以上。目前這是不可能的,但是人們認識到[ubyte]向量上的(force_align: n)屬性可能是有用的,但也可以通過嵌套的flatbuffers來處理,該flatbuffer也可以對準數據。
在0.6.0中引入的固定長度陣列允許包含固定長度標量,結構和字符的陣列的結構。陣列像相似類型的向量一樣解析,但如果比預期較短,則填充為零,並且如果比預期的時間更長。如果陣列比預期的,而不是零填充,則標誌reject_array_underflow將錯誤。 FLAG skip_array_overflow將允許延長的數組,並簡單地刪除額外的元素。
char陣列像字符串一樣被解析,如果短於預期的話,則零填充量,但並非終止零。像“ Hello”這樣的字符串將完全適合[char:5]類型的字段。未打印尾部零字符,但嵌入式零字符是。這允許無損失的往返行駛,而無需零墊串。請注意,其他陣列總是完全打印。如果設置了標誌skip_array_overflow ,則可能在多字節字符的中間截斷字符串。驗證者未檢查或執行此操作。
打印機和解析器都有能力接受修改其行為的運行時標誌。請參閱標題文件註釋以獲取文檔和測試用例以獲取示例。值得注意的是,可以在解析時打印無引用的符號並忽略未知字段而不是生成錯誤。
請注意,不棄用的字段被認為是解析過程中未知字段,因此為了從舊的模式版本中處理帶有棄用字段的舊架構,必須跳過未知的符號。
截至v0.5.1 test_json.c,演示瞭如何使用單個解析器驅動程序來解析不同的表類型,而無需更改驅動程序或模式。
例如,以下佈局可用於配置通用解析器或打印機。
struct json_scope {
const char *identifier;
flatcc_json_parser_table_f *parser;
flatcc_json_printer_table_f *printer;
flatcc_table_verifier_f *verifier;
};
static const struct json_scope Monster = {
/* The is the schema global file identifier. */
ns(Monster_identifier),
ns(Monster_parse_json_table),
ns(Monster_print_json_table),
ns(Monster_verify_table)
};
現在,駕駛員可以使用Monster範圍或根據需要替換新範圍:
/* Abbreviated ... */
struct json_scope = Monster;
flatcc_json_parser_table_as_root(B, &parser_ctx, json, strlen(json), parse_flags,
scope->identifier, scope->parser);
/* Printing and verifying works roughly the same. */
生成的表MyGame_Example_Monster_parse_json_as_root是一個薄的便利包裝器,大致實現了上述。
生成的monster_test_parse_json是一個更高級別的便利包裝器,名為架構文件本身,而不是任何特定表。它解析了在架構中配置的root_type 。這就是test_json.c測試驅動程序在v0.5.1之前操作的方式,但很難測試解析和打印不同的表類型。
請注意,JSON解析並不需要驗證,因為生成的JSON解析器應該構建始終驗證的緩衝區(除了二進制編碼的嵌套緩衝區除外),但對測試很有用。
請注意,JSON解析和打印非常快地達到500MB/s的打印,約300 MB/s用於解析。浮點解析可能會偏向這些數字。整數和浮點解析和打印是通過便攜式庫中的支持功能來處理的。另外,浮點include/flatcc/portable/grisu3_*庫將使用,除非通過編譯時標誌明確禁用。禁用grisu3將恢復為sprintf和strtod 。在某些罕見的特殊情況下,grisu3將倒入strtod和grisu3 。由於對strtod的依賴,並且由於strtod無法有效處理非零終止的緩衝區,因此建議將零終止緩衝液進行。另外,可以用允許轉換錯誤的標誌來編譯Grisu3。這些錯誤非常小,仍然正確,但可能會打破一些校驗和檢查。允許這些錯誤可以顯著提高解析速度,並將基準從低於100萬的解析移至每秒以上的700個字節JSON弦,以2.2 GHz CORE-I7的速度移動到每秒以上。
儘管由於緊湊的尺寸,但未引用的字符串聽起來可能更有效,但實際上它的處理速度較慢。此外,大型FlatBuffer生成的JSON文件可以使用GZIP或使用LZ4的因子4壓縮一個因子8,因此這可能是優化的更好場所。對於小型緩衝區,壓縮flatbuffer二進製文件可能更有效,但是對於大型文件,由於格式缺乏指針,JSON實際上可能會更大的壓縮。
SSE 4.2已通過實驗添加,但是收益是有限的,因為它在解析空間時效果最好,並且空間解析已經很快沒有第4.2 sse 4.2,而且如果急忙,它可能會拋棄空間。對於解析字符串,對於第4.2 sse 4.2弦掃描的瑣碎使用不能很好地工作,因為必須檢測到ASCII 32下方的所有逃生代碼,而不僅僅是搜索 and " 。這並不是說沒有收益,它們似乎並不值得。
對64位的解析器非常優化,因為它直接在代碼中實現了8字節寬的Trie。它可能也適用於32位編譯器,但尚未進行測試。大型Trie確實在編譯時間造成了一些壓力。優化-O2超越-O2會導致太大的二進制方法,以抵消任何速度增長。
架構中包含的屬性在全局名稱空間中查看,每個屬性添加到此名稱空間中,因此模式文件可以使用包含的屬性,而沒有名稱空間前綴。
每個包含的模式還將在全局範圍中添加類型,直到看到namespace聲明為止。隨附的架構不會繼承包含文件或更早的文件的名稱空間,因此所有架構文件都始於全局範圍。但是,隨附的文件可以查看以前在全局範圍中定義的其他類型。因為包括的語句總是首先出現在架構中,所以只能較早地包含文件,而不是包含架構的類型。
任何隨附的模式的生成的輸出都不符合其包含的方式,但是如果沒有較早的文件存在並首先包含在內,則可能不會編譯。通過包括高級myschema.h或myschema_builder.h所有這些依賴項都正確處理。
注意: libflatcc.a只能在給出架構作為內存緩衝區時解析單個架構,但是在給出文件名時可以處理上述情況。可以連接架構文件,但namespace;聲明必須作為分隔符插入,以便在每個包含文件的開始時還原為全局名稱空間。這可能會導致細微的錯誤,因為如果一個父架模板包括兩個子模架a.fbs和b.fbs ,那麼b.fbs即使共享名稱空間,也應該在a.fbs中看到任何東西。這在實踐中很少是一個問題,但這意味著內存緩衝區的架構編譯無法創作驗證模式。必須隔離架構的原因是,否則給定模式的代碼生成可能會隨著其使用方式而變化,從而導致用戶代碼中的錯誤錯誤。
如果需要一個字段,例如monster.name,則表結束呼叫將在調試模式下斷言,並在非刪除構建中創建不正確的表。當庫代碼中發生的那樣,斷言它可能並不容易,並且不會告訴丟失哪個字段。
在閱讀名稱時,調試模式將再次斷言,非刪除構建將返回默認值。
編寫相同的字段兩次也將觸發調試構建中的斷言。
緩衝區可通過使用以根係為根來創建緩衝區的能力來用於高速通信。此外,默認的發射極支持flatcc_emitter_direct_buffer用於小型緩衝區,因此不需要額外的複制步驟即可在內存中獲得線性緩衝區。初步測量表明,由於構建器對象必須在涉及零分配的緩衝液的緩衝區之間重置,因此該速度的速度限制(約6-7磨。緩衝液/秒)是有限制的。具有簡單矢量的小表達到了大約一半的速度。對於真正高速的結構,需要專門的建築商。另請參見Monster_test.c。
存儲在緩衝區中的所有類型都有類型的後綴,例如Monster_table_t或Vec3_struct_t (以及我們在這裡遺漏的名稱空間前綴)。這些類型僅讀取到Endian編碼的數據中。枚舉類型只是從生成的代碼中輕鬆掌握的常數。桌子很密集,因此它們永遠不會直接訪問。
枚舉支持架構進化,這意味著可以將更多名稱添加到未來模式版本中的枚舉中。從v0.5.0開始,可以使用該函數_is_known_value檢查當前架構版本是否已知枚舉值。
結構具有雙重目的,因為它們也是本機格式的有效類型,但本機代替的目的略有不同。因此,約定是指在FlatBuffer中編碼的結構的const指針具有類型的Vec3_struct_t ,其中作為可寫入的本機結構指針具有Vec3_t *或struct Vec3 *的類型。
所有類型都有_vec_t後綴,該後綴是指向基礎類型的const指針。例如, Monster_table_t具有向量類型Monster_vec_t 。還有一個不使用後綴_mutable_vec_t的非const變體,很少使用。但是,可以在緩衝區中對向量進行排序,為此,必須先將向量施加到可變的。 A vector (or string) type points to the element with index 0 in the buffer, just after the length field, and it may be cast to a native type for direct access with attention to endian encoding. (Note that table_t types do point to the header field unlike vectors.) These types are all for the reader interface. Corresponding types with a _ref_t suffix such as _vec_ref_t are used during the construction of buffers.
Native scalar types are mapped from the FlatBuffers schema type names such as ubyte to uint8_t and so forth. These types also have vector types provided in the common namespace (default flatbuffers_ ) so a [ubyte] vector has type flatbuffers_uint8_vec_t which is defined as const uint8_t * .
The FlatBuffers boolean type is strictly 8 bits wide so we cannot use or emulate <stdbool.h> where sizeof(bool) is implementation dependent. Therefore flatbuffers_bool_t is defined as uint8_t and used to represent FlatBuffers boolean values and the constants of same type: flatbuffers_true = 1 and flatbuffers_false = 0 . Even so, pstdbool.h is available in the include/flatcc/portable directory if bool , true , and false are desired in user code and <stdbool.h> is unavailable.
flatbuffers_string_t is const char * but imply the returned pointer has a length prefix just before the pointer. flatbuffers_string_vec_t is a vector of strings. The flatbufers_string_t type guarantees that a length field is present using flatbuffers_string_len(s) and that the string is zero terminated. It also suggests that it is in utf-8 format according to the FlatBuffers specification, but not checks are done and the flatbuffers_create_string(B, s, n) call explicitly allows for storing embedded null characters and other binary data.
All vector types have operations defined as the typename with _vec_t replaced by _vec_at and _vec_len . For example flatbuffers_uint8_vec_at(inv, 1) or Monster_vec_len(inv) . The length or _vec_len will be 0 if the vector is missing whereas _vec_at will assert in debug or behave undefined in release builds following out of bounds access. This also applies to related string operations.
The FlatBuffers schema uses the following scalar types: ubyte , byte , ushort , short, uint , int , ulong , and long to represent unsigned and signed integer types of length 8, 16, 32, and 64 respectively. The schema syntax has been updated to also support the type aliases uint8 , int8 , uint16 , int16 , uint32 , int32 , uint64 , int64 to represent the same basic types. Likewise, the schema uses the types float and double to represent IEEE-754 binary32 and binary64 floating point formats where the updated syntax also supports the type aliases float32 and float64 .
The C interface uses the standard C types such as uint8 and double to represent scalar types and this is unaffected by the schema type name used, so the schema vector type [float64] is represented as flatbuffers_double_vec_t the same as [double] would be.
Note that the C standard does not guarantee that the C types float and double are represented by the IEEE-754 binary32 single precision format and the binary64 double precision format respectively, although they usually are. If this is not the case FlatCC cannot work correctly with FlatBuffers floating point values. (If someone really has this problem, it would be possible to fix).
Unions are represented with a two table fields, one with a table field and one with a type field. See separate section on Unions. As of flatcc v0.5.0 union vectors are also supported.
A union represents one of several possible tables. A table with a union field such as Monster.equipped in the samples schema will have two accessors: MyGame_Sample_Monster_equipped(t) of type flatbuffers_generic_t and MyGame_Sample_Monster_equipped_type(t) of type MyGame_Sample_Equipment_union_type_t . A generic type is is just a const void pointer that can be assigned to the expected table type, struct type, or string type. The enumeration has a type code for member of the union and also MyGame_Sample_Equipment_NONE which has the value 0.
The union interface were changed in 0.5.0 and 0.5.1 to use a consistent { type, value } naming convention for both unions and union vectors in all interfaces and to support unions and union vectors of multiple types.
A union can be accessed by its field name, like Monster MyGame_Sample_Monster_equipped(t) and its type is given by MyGame_Sample_Monster_type(t) , or a flatbuffers_union_t struct can be returned with MyGame_Sample_monster_union(t) with the fields { type, value }. A union vector is accessed in the same way but { type, value } represents a type vector and a vector of the given type, eg a vector Monster tables or a vector of strings.
There is a test in monster_test.c covering union vectors and a separate test focusing on mixed type unions that also has union vectors.
Googles monster_test.fbs schema has the union (details left out):
namespace MyGame.Example2;
table Monster{}
namespace MyGame.Example;
table Monster{}
union Any { Monster, MyGame.Example2.Monster }
where the two Monster tables are defined in separate namespaces.
flatcc rejects this schema due to a name conflict because it uses the basename of a union type, here Monster to generate the union member names which are also used in JSON parsing. This can be resolved by adding an explicit name such as Monster2 to resolve the conflict:
union Any { Monster, Monster2: MyGame.Example2.Monster }
This syntax is accepted by both flatc and flatcc .
Both versions will implement the same union with the same type codes in the binary format but generated code will differ in how the types are referred to.
In JSON the monster type values are now identified by MyGame.Example.Any.Monster , or just Monster , when assigning the first monster type to an Any union field, and MyGame.Example.Any.Monster2 , or just Monster2 when assigning the second monster type. C uses the usual enum namespace prefixed symbols like MyGame_Example_Any_Monster2 .
Fixed Length Arrays is a late feature to the FlatBuffers format introduced in flatc and flatcc mid 2019. Currently only scalars arrays are supported, and only as struct fields. To use fixed length arrays as a table field wrap it in a struct first. It would make sense to support struct elements and enum elements, but that has not been implemented. Char arrays are more controversial due to verification and zero termination and are also not supported. Arrays are aligned to the size of the first field and are equivalent to repeating elements within the struct.
The schema syntax is:
struct MyStruct {
my_array : [float:10];
}
See test_fixed_array in monster_test.c for an example of how to work with these arrays.
Flatcc opts to allow arbitrary length fixed length arrays but limit the entire struct to 2^16-1 bytes. Tables cannot hold larger structs, and the C language does not guarantee support for larger structs. Other implementations might have different limits on maximum array size. Arrays of 0 length are not permitted.
Optional scalar table fields were introduced to FlatBuffers mid 2020 in order to better handle null values also for scalar data types, as is common in SQL databases. Before describing optional values, first understand how ordinary scalar values work in FlatBuffers:
Imagine a FlatBuffer table with a mana field from the monster sample schema. Ordinarily a scalar table field has implicit default value of 0 like mana : uint8; , or an explicit default value specified in the schema like mana : uint8 = 100; . When a value is absent from a table field, the default value is returned, and when a value is added during buffer construction, it will not actually be stored if the value matches the default value, unless the force_add option is used to write a value even if it matches the default value. Likewise the is_present method can be used to test if a field was actually stored in the buffer when reading it.
When a table has many fields, most of which just hold default settings, signficant space can be saved using default values, but it also means that an absent value does not indicate null. Field absence is essentially just a data compression technique, not a semantic change to the data. However, it is possible to use force_add and is_present to interpret values as null when not present, except that this is not a standardized technique. Optional fields represents a standardized way to achieve this.
Scalar fields can be marked as optional by assigning null as a default value. For example, some objects might not have a meaningful mana value, so it could be represented as lifeforce : uint8 = null . Now the lifeforce field has become an optional field. In the FlatCC implementation this means that the field is written, it will always be written also if the value is 0 or any other representable value. It also means that the force_add method is not available for the field because force_add is essentially always in effect for the field. On the read side, optional scalar fields behave exactly is ordinary scalar fields that have not specified a default value, that is, if the field is absent, 0 will be returned and is_present will return false. Instead optional scalar fields get a new accessor method with the suffix _option() which returns a struct with two fiels: { is_null, value } where _option().is_null == !is_present() and _option().value is the same value is the _get() method, which will be 0 if is_null is true. The option struct is named after the type similar to unions, for example flatbuffers_uint8_option_t or MyGame_Example_Color_option_t , and the option accessor method also works similar to unions. Note that _get() will also return 0 for optional enum values that are null (ie absent), even if the enum value does not have an enumerated element with the value 0. Normally enums without a 0 element is not allowed in the schema unless a default value is specified, but in this case it is null, and _get() needs some value to return in this case.
By keeping the original accessors, read logic can be made simpler and faster when it is not important whether a value is null or 0 and at the same time the option value can be returned and stored.
Note that struct fields cannot be optional. Also note that, non-scalar table fields are not declared optional because these types can already represent null via a null pointer or a NONE union type.
JSON parsing and printing change behavior for scalar fields by treating absent fields differently according the optional semantics. For example parsing a missing field will not store a default value even if the parser is executed with a flag to force default values to be stored and the printer will not print absent optional fields even if otherwise flagged to print default values. Currenlty the JSON printers and parsers do not print or parse JSON null and can only represent null be absence of a field.
For an example of reading and writing, as well as printing and parsing JSON, optional scalar fields, please refer to optional_scalars_test.fbs and optional_scalars_test.c.
The pendian_detect.h` file detects endianness for popular compilers and provides a runtime fallback detection for others. In most cases even the runtime detection will be optimized out at compile time in release builds.
The FLATBUFFERS_LITTLEENDIAN flag is respected for compatibility with Googles flatc compiler, but it is recommended to avoid its use and work with the mostly standard flags defined and/or used in pendian_detect.h , or to provide for additional compiler support.
As of flatcc 0.4.0 there is support for flatbuffers running natively on big endian hosts. This has been tested on IBM AIX. However, always run tests against the system of interest - the release process does not cover automated tests on any BE platform.
As of flatcc 0.4.0 there is also support for compiling the flatbuffers runtime library with flatbuffers encoded in big endian format regardless of the host platforms endianness. Longer term this should probably be placed in a separate library with separate name prefixes or suffixes, but it is usable as is. Redefine FLATBUFFERS_PROTOCOL_IS_LE/BE accordingly in flatcc_types.h. This is already done in the be branch. This branch is not maintained but the master branch can be merged into it as needed.
Note that standard flatbuffers are always encoded in little endian but in situations where all buffer producers and consumers are big endian, the non standard big endian encoding may be faster, depending on intrinsic byteswap support. As a curiosity, the load_test actually runs faster with big endian buffers on a little endian MacOS platform for reasons only the optimizer will know, but read performance of small buffers drop to 40% while writing buffers generally drops to 80-90% performance. For platforms without compiler intrinsics for byteswapping, this can be much worse.
Flatbuffers encoded in big endian will have the optional file identifier byteswapped. The interface should make this transparent, but details are still being worked out. For example, a buffer should always verify the monster buffer has the identifier "MONS", but internally the buffer will store the identifier as "SNOM" on big endian encoded buffers.
Because buffers can be encode in two ways, flatcc uses the term native endianness and protocol endianess. _pe is a suffix used in various low level API calls to convert between native and protocol endianness without caring about whether host or buffer is little or big endian.
If it is necessary to write application code that behaves differently if the native encoding differs from protocol encoding, use flatbuffers_is_pe_native() . This is a function, not a define, but for all practical purposes it will have same efficience while also supporting runtime endian detection where necessary.
The flatbuffer environment only supports reading either big or little endian for the time being. To test which is supported, use the define FLATBUFFERS_PROTOCOL_IS_LE or FLATBUFFERS_PROTOCOL_IS_BE . They are defines as 1 and 0 respectively.
The builder API often returns a reference or a pointer where null is considered an error or at least a missing object default. However, some operations do not have a meaningful object or value to return. These follow the convention of 0 for success and non-zero for failure. Also, if anything fails, it is not safe to proceed with building a buffer. However, to avoid overheads, there is no hand holding here. On the upside, failures only happen with incorrect use or allocation failure and since the allocator can be customized, it is possible to provide a central error state there or to guarantee no failure will happen depending on use case, assuming the API is otherwise used correctly. By not checking error codes, this logic also optimizes out for better performance.
The builder API does not support sorting due to the complexity of customizable emitters, but the reader API does support sorting so a buffer can be sorted at a later stage. This requires casting a vector to mutable and calling the sort method available for fields with keys.
The sort uses heap sort and can sort a vector in-place without using external memory or recursion. Due to the lack of external memory, the sort is not stable. The corresponding find operation returns the lowest index of any matching key, or flatbuffers_not_found .
When configured in config.h (the default), the flatcc compiler allows multiple keyed fields unlike Googles flatc compiler. This works transparently by providing <table_name>_vec_sort_by_<field_name> and <table_name>_vec_find_by_<field_name> methods for all keyed fields. The first field maps to <table_name>_vec_sort and <table_name>_vec_find . Obviously the chosen find method must match the chosen sort method. The find operation is O(logN).
As of v0.6.0 the default key used for find and and sort without the by_name suffix is the field with the smaller id instead of the first listed in the schema which is often but not always the same thing.
v0.6.0 also introduces the primary_key attribute that can be used instead of the key attribute on at most one field. The two attributes are mutually exclusive. This can be used if a key field with a higher id should be the default key. There is no difference when only one field has a key or primary_key attribute, so in that case choose key for compatiblity. Googles flatc compiler does not recognize the primary_key attribute.
As of v0.6.0 a 'sorted' attribute has been introduced together with the sort operations <table_name>_sort and <union_name>_sort . If a table or a union, directly or indirectly, contains a vector with the 'sorted' attribute, then the sort operation is made available. The sort will recursively visit all children with vectors marked sorted. The sort operatoin will use the default (primary) key. A table or union must first be cast to mutable, for example ns(Monster_sort((ns(Monster_mutable_table_t))monster) . The actual vector sort operations are the same as before, they are just called automatically. The sorted attribute can only be set on vectors that are not unions. The vector can be of scalar, string, struct, or table type. sorted is only valid for a struct or table vector if the struct or table has a field with a key or primary_key attribute. NOTE: A FlatBuffer can reference the same object multiple times. The sort operation will be repeated if this is the case. Sometimes that is OK, but if it is a concern, remove the sorted attribute and sort the vector manually. Note that sharing can also happen via a shared containing object. The sort operations are generated in _reader.h files and only for objects directly or indirectly affected by the sorted attribute. Unions have a new mutable case operator for use with sorting unions: ns(Any_sort(ns(Any_mutable_cast)(my_any_union)) . Usually unions will be sorted via a containing table which performs this cast automatically. See also test_recursive_sort in monster_test.c.
As of v0.4.1 <table_name>_vec_scan_by_<field_name> and the default <table_name>_vec_scan are also provided, similar to find , but as a linear search that does not require the vector to be sorted. This is especially useful for searching by a secondary key (multiple keys is a non-standard flatcc feature). _scan_ex searches a sub-range [a, b) where b is an exclusive index. b = flatbuffers_end == flatbuffers_not_found == (size_t)-1 may be used when searching from a position to the end, and b can also conveniently be the result of a previous search.
rscan searches in the opposite direction starting from the last element. rscan_ex accepts the same range arguments as scan_ex . If a >= b or a >= len the range is considered empty and flatbuffers_not_found is returned. [r]scan[_ex]_n[_by_name] is for length terminated string keys. See monster_test.c for examples.
Note that find requires key attribute in the schema. scan is also available on keyed fields. By default flatcc will also enable scan by any other field but this can be disabled by a compile time flag.
Basic types such as uint8_vec also have search operations.
See also Builder Interface Reference and monster_test.c.
The FlatBuffers format does not fully distinguish between default values and missing or null values but it is possible to force values to be written to the buffer. This is discussed further in the Builder Interface Reference. For SQL data roundtrips this may be more important that having compact data.
The _is_present suffix on table access methods can be used to detect if value is present in a vtable, for example Monster_hp_present . Unions return true of the type field is present, even if it holds the value None.
The add methods have corresponding force_add methods for scalar and enum values to force storing the value even if it is default and thus making it detectable by is_present .
The portable library is placed under include/flatcc/portable and is required by flatcc, but isn't strictly part of the flatcc project. It is intended as an independent light-weight header-only library to deal with compiler and platform variations. It is placed under the flatcc include path to simplify flatcc runtime distribution and to avoid name and versioning conflicts if used by other projects.
The license of portable is different from flatcc . It is mostly MIT or Apache depending on the original source of the various parts.
A larger set of portable files is included if FLATCC_PORTABLE is defined by the user when building.
cc -D FLATCC_PORTABLE -I include monster_test.c -o monster_test
Otherwise a targeted subset is included by flatcc_flatbuffers.h in order to deal with non-standard behavior of some C11 compilers.
pwarnings.h is also always included so compiler specific warnings can be disabled where necessary.
The portable library includes the essential parts of the grisu3 library found in external/grisu3 , but excludes the test cases. The JSON printer and parser relies on fast portable numeric print and parse operations based mostly on grisu3.
If a specific platform has been tested, it would be good with feedback and possibly patches to the portability layer so these can be made available to other users.
Note: if a test fails, see Strict Aliasing for a possible resolution.
To initialize and run the build (see required build tools below):
scripts/build.sh
The bin and lib folders will be created with debug and release build products.
The build depends on CMake . By default the Ninja build tool is also required, but alternatively make can be used.
Optionally switch to a different build tool by choosing one of:
scripts/initbuild.sh make
scripts/initbuild.sh make-concurrent
scripts/initbuild.sh ninja
where ninja is the default and make-concurrent is make with the -j flag.
To enforce a 32-bit build on a 64-bit machine the following configuration can be used:
scripts/initbuild.sh make-32bit
which uses make and provides the -m32 flag to the compiler. A custom build configuration X can be added by adding a scripts/build.cfg.X file.
scripts/initbuild.sh cleans the build if a specific build configuration is given as argument. Without arguments it only ensures that CMake is initialized and is therefore fast to run on subsequent calls. This is used by all test scripts.
To install build tools on OS-X, and build:
brew update
brew install cmake ninja
git clone https://github.com/dvidelabs/flatcc.git
cd flatcc
scripts/build.sh
To install build tools on Ubuntu, and build:
sudo apt-get update
sudo apt-get install cmake ninja-build
git clone https://github.com/dvidelabs/flatcc.git
cd flatcc
scripts/build.sh
To install build tools on Centos, and build:
sudo yum group install "Development Tools"
sudo yum install cmake
git clone https://github.com/dvidelabs/flatcc.git
cd flatcc
scripts/initbuild.sh make # there is no ninja build tool
scripts/build.sh
OS-X also has a HomeBrew package:
brew update
brew install flatcc
or for the bleeding edge:
brew update
brew install flatcc --HEAD
Install CMake, MSVC, and git (tested with MSVC 14 2015).
In PowerShell:
git clone https://github.com/dvidelabs/flatcc.git
cd flatcc
mkdir buildMSVC
cd buildMSVC
cmake -G "Visual Studio 14 2015" ....
Optionally also build from the command line (in buildMSVC):
cmake --build . --target --config Debug
cmake --build . --target --config Release
In Visual Studio:
open flatccbuildMSVCFlatCC.sln
build solution
choose Release build configuration menu
rebuild solution
Note that flatccCMakeList.txt sets the -DFLATCC_PORTABLE flag and that includeflatccportablepwarnings.h disable certain warnings for warning level -W3.
Docker圖像:
Users have been reporting some degree of success using cross compiles from Linux x86 host to embedded ARM Linux devices.
For this to work, FLATCC_TEST option should be disabled in part because cross-compilation cannot run the cross-compiled flatcc tool, and in part because there appears to be some issues with CMake custom build steps needed when building test and sample projects.
2024-03-08: WARNING: -O2 -mcpu=cortex-m7 targets using the arm-none-eabi 13.2.Rel1 toolchain can result in uninitialized stack access when not compiled with -fno-strict-aliasing -mcpu=cortex-m0 and -mcpu=cortex-m1 appears to be unaffected. See also issue #274. 2024-10-03: Fix available on flatcc master branch when you read this. See also CHANGELOG comments for release 0.6.2.
The option FLATCC_RTONLY will disable tests and only build the runtime library.
The following is not well tested, but may be a starting point:
mkdir -p build/xbuild
cd build/xbuild
cmake ../.. -DBUILD_SHARED_LIBS=on -DFLATCC_RTONLY=on
-DCMAKE_BUILD_TYPE=Release
Overall, it may be simpler to create a separate Makefile and just compile the few src/runtime/*.c into a library and distribute the headers as for other platforms, unless flatcc is also required for the target. Or to simply include the runtime source and header files in the user project.
Note that no tests will be built nor run with FLATCC_RTONLY enabled. It is highly recommended to at least run the tests/monster_test project on a new platform.
Some target systems will not work with Posix malloc , realloc , free and C11 aligned_alloc . Or they might, but more allocation control is desired. The best approach is to use flatcc_builder_custom_init to provide a custom allocator and emitter object, but for simpler case or while piloting a new platform flatcc_alloc.h can be used to override runtime allocation functions. Carefully read the comments in this file if doing so. There is a test case implementing a new emitter, and a custom allocator can be copied from the one embedded in the builder library source.
On systems where the default POSIX assert call is unavailable, or when a different assert behaviour is desirable, it is possible to override the default behaviour in runtime part of flatcc library via logic defined in flatcc_assert.h.
By default Posix assert is beeing used. It can be changed by preprocessor definition:
-DFLATCC_ASSERT=own_assert
but it will not override assertions used in the portable library, notably the Grisu3 fast numerical conversion library used with JSON parsing.
Runtime assertions can be disabled using:
-DFLATCC_NO_ASSERT
This will also disable Grisu3 assertions. See flatcc_assert.h for details.
The <assert.h> file will in all cases remain a dependency for C11 style static assertions. Static assertions are needed to ensure the generated structs have the correct physical layout on all compilers. The portable library has a generic static assert implementation for older compilers.
By default libraries are built statically.
Occasionally there are requests #42 for also building shared libraries. It is not clear how to build both static and shared libraries at the same time without choosing some unconvential naming scheme that might affect install targets unexpectedly.
CMake supports building shared libraries out of the box using the standard library name using the following option:
CMAKE ... -DBUILD_SHARED_LIBS=ON ...
See also CMake Gold: Static + shared.
The Flatcc build files should take care of strict aliasing issues on common platforms, but it is not a solved problem, so here is some background information.
In most cases this is a non-issue with the current flatcc code base, but that does not help in the cases where it is an issue.
Compilers have become increasingly aggressive with applying, and defaulting to, strict aliasing rules.
FlatCC does not guarantee that strict aliasing rules are followed, but the code base is updated as issues are detected. If a test fails or segfaults the first thing to check is -fno-strict-aliasing , or the platform equivalent, or to disable pointer casts, as discussed below.
Strict aliasing means that a cast like p2 = *(T *)p1 is not valid because the compiler thinks that p2 does not depend on data pointed to by p1. In most cases compilers are sensible enough to handle this, but not always. It can, and will, lead to reading from uninitialized memory or segfaults. There are two ways around this, one is to use unions to convert from integer to float, which is valid in C, but not in C++, and the other is to use memcpy for small constant sizes, which is guaranteed safe, but can be slow if not optimized, and it is not always optimized. (Not strictly memcpy but access via cast to char * or other "narrow" type).
FlatCC manages this in flatcc_accessors.h which forwards to platform dependent code in pmemaccess.h. Note that is applies to the runtime code base only. For compile time the only issue should be hash tables and these should also be safe.
FlatCC either uses optimized memcpy or non-compliant pointer casts depending on the platform. Essentially, buffer memory is first copied, or pointer cast, into an unsigned integer of a given size. This integer is then endian converted into another unsigned integer. Then that integer is converted into a final integer type or floating point type using union casts. This generally optimizes out to very few assembly instructions, but when it does not, code size and execution time can grow significantly.
It has been observed that targets both default to strict aliasing with -O2 optimization, and at the same to uses a function call for memcpy(dest, src, sizeof(uint32_t)) , but where __builtin_memcpy does optimize well, hence requiring detection of a fast memcpy operation.
This is a game between being reasonably performant and compliant.
-DPORTABLE_MEM_PTR_ACCESS=0 will force the runtime code to not use pointer casts but it can potentially generate suboptimal code and can be set 1 if the compiler and build configuration is known to not have issues with strict aliasing. It is set to 1 for most x86/64 targets since this has been working for a long time in FlatCC builds and tests, while memcpy might not work efficient.
Install targes may be built with:
mkdir -p build/install
cd build/install
cmake ../.. -DBUILD_SHARED_LIBS=on -DFLATCC_RTONLY=on
-DCMAKE_BUILD_TYPE=Release -DFLATCC_INSTALL=on
make install
However, this is not well tested and should be seen as a starting point. The normal scripts/build.sh places files in bin and lib of the source tree.
By default lib files a built into the lib subdirectory of the project. This can be changed, for example like -DFLATCC_INSTALL_LIB=lib64 .
To distribute the compiled binaries the following files are required:
編譯器:
bin/flatcc (command line interface to schema compiler)
lib/libflatcc.a (optional, for linking with schema compiler)
include/flatcc/flatcc.h (optional, header and doc for libflatcc.a)
運行時:
include/flatcc/** (runtime header files)
include/flatcc/reflection (optional)
include/flatcc/support (optional, only used for test and samples)
lib/libflatccrt.a (runtime library)
In addition the runtime library source files may be used instead of libflatccrt.a . This may be handy when packaging the runtime library along with schema specific generated files for a foreign target that is not binary compatible with the host system:
src/runtime/*.c
The build products from MSVC are placed in the bin and lib subdirectories:
flatccbinDebugflatcc.exe
flatcclibDebugflatcc_d.lib
flatcclibDebugflatccrt_d.lib
flatccbinReleaseflatcc.exe
flatcclibReleaseflatcc.lib
flatcclibReleaseflatccrt.lib
Runtime includeflatcc directory is distributed like other platforms.
跑步
scripts/test.sh [--no-clean]
NOTE: The test script will clean everything in the build directy before initializing CMake with the chosen or default build configuration, then build Debug and Release builds, and run tests for both.
The script must end with TEST PASSED , or it didn't pass.
To make sure everything works, also run the benchmarks:
scripts/benchmark.sh
In Visual Studio the test can be run as follows: first build the main project, the right click the RUN_TESTS target and chose build. See the output window for test results.
It is also possible to run tests from the command line after the project has been built:
cd buildMSVC
ctest
Note that the monster example is disabled for MSVC 2010.
Be aware that tests copy and generate certain files which are not automatically cleaned by Visual Studio. Close the solution and wipe the MSVC directory, and start over to get a guaranteed clean build.
Please also observe that the file .gitattributes is used to prevent certain files from getting CRLF line endings. Using another source control systems might break tests, notably test/flatc_compat/monsterdata_test.golden .
Note: Benchmarks have not been ported to Windows.
配置
config/config.h
drives the permitted syntax and semantics of the schema compiler and code generator. These generally default to be compatible with Googles flatc compiler. It also sets things like permitted nesting depth of structs and tables.
The runtime library has a separate configuration file
include/flatcc/flatcc_rtconfig.h
This file can modify certain aspects of JSON parsing and printing such as disabling the Grisu3 library or requiring that all names in JSON are quoted.
For most users, it should not be relevant to modify these configuration settings. If changes are required, they can be given in the build system - it is not necessary to edit the config files, for example to disable trailing comma in the JSON parser:
cc -DFLATCC_JSON_PARSE_ALLOW_TRAILING_COMMA=0 ...
The compiler library libflatcc.a can compile schemas provided in a memory buffer or as a filename. When given as a buffer, the schema cannot contain include statements - these will cause a compile error.
When given a filename the behavior is similar to the commandline flatcc interface, but with more options - see flatcc.h and config/config.h .
libflatcc.a supports functions named flatcc_... . reflection... may also be available which are simple the C generated interface for the binary schema. The builder library is also included. These last two interfaces are only present because the library supports binary schema generation.
The standalone runtime library libflatccrt.a is a collection of the src/runtime/*.c files. This supports the generated C headers for various features. It is also possible to distribute and compile with the source files directly. For debugging, it is useful to use the libflatccrt_d.a version because it catches a lot of incorrect API use in assertions.
The runtime library may also be used by other languages. See comments in flatcc_builder.h. JSON parsing is on example of an alternative use of the builder library so it may help to inspect the generated JSON parser source and runtime source.
Mostly for implementers: FlatBuffers Binary Format
See Security Considerations.
FlatCC coding style is largely similar to the WebKit Style, with the following notable exceptions:
<stdint.h> types are made available.if (err) return -1; .0 ./* A comment. */true and false keywords are not used (pre C99).snake_case is used over camelCase .#pragma once because it is non-standard and not always reliable in filesystems with ambigious paths.config.h inclusion might be handled differently in that flatbuffers.h includes the config file.unsigned is not used without int for historical reasons. Generally a type like uint32_t is preferred.TODO: instead of FIXME: in comments for historical reasons.All the main source code in compiler and runtime aim to be C11 compatible and uses many C11 constructs. This is made possible through the included portable library such that older compilers can also function. Therefore any platform specific adaptations will be provided by updating the portable library rather than introducing compile time flags in the main source code.
See Benchmarks