Model overview
SEC-BERT is a family of BERT models for the financial domain, intended to assist financial NLP research and FinTech applications. SEC-BERT consists of the following models: * **SEC-BERT-BASE** (this model): Same architecture as BERT-BASE trained on financial documents. * **SEC-BERT-NUM**: Same as SEC-BERT-BASE but we replace every number token with a [NUM] pseudo-token handling all numeric expressions in a uniform manner, disallowing their fragmentation * **SEC-BERT-SHAPE**: Same as SEC-BERT-BASE but we replace numbers with pseudo-tokens that represent the number’s shape, so numeric expressions (of known shapes) are no longer fragmented, e.g., '53.2' becomes '[XX.X]' and '40,200.5' becomes '[XX,XXX.X]'. The model was pre-trained on 260,773 10-K filings from 1993-2019, publicly available at U.S. Securities and Exchange Commission (SEC) * We created a new vocabulary of 30k subwords by training a BertWordPieceTokenizer from scratch on the pre-training corpus. * We trained BERT using the official code provided in Google BERT's GitHub repository . * We then used Hugging Face's Transformers conversion script to convert the TF checkpoint in the desired format in order to be able to load the model in two lines of code for both PyTorch and TF2 users. * We release a model similar to the English BERT-BASE model (12-layer, 768-hidden, 12-heads, 110M parameters). * We chose to follow the same training set-up: 1 million training steps with batches of 256 sequences of length 512 with an initial learning rate 1e-4. * We were able to use a single Google Cloud TPU v3-8 provided for free from TensorFlow Research Cloud (TRC), while also utilizing GCP research credits. Huge thanks to both Google programs for supporting us!
Key features
特点摘要待依据来源补齐，不凭模型名称推断能力。
Limitations & considerations
Sources are insufficient to establish specific limitations. Verify task performance, latency, data policies, licensing and version lifecycle.
Source prices & cost analysis
Prices vary by channel, region and configuration. Channel quotes are listed independently and are not treated as a single official provider price.
| Source / channel | Input | Output | Billing unit | Collected at |
|---|---|---|---|---|
| No verified prices yet. Check the source channel. | ||||
Estimate monthly base token costs
Estimates exclude cache discounts, tiered pricing, multimodal surcharges, web search, taxes and additional reasoning output.
Public benchmarks & run configuration
暂无已采集的可核验公开评分。模型未上榜不代表能力较低。
View full indexed rankings →Frequently asked questions
Do open weights permit free commercial use?
Not necessarily. Check the version license, commercial conditions and usage policies. Self-hosting incurs compute and maintenance costs.
Are these real-time billing prices?
These are timestamped source snapshots, not live billing prices. Check the channel for current versions, regions, discounts and billing methods.
Sources & traceability
许可证：cc-by-sa-4.0。模型介绍与第三方评测均以原来源的具体版本和方法为准。