Skip to content

Commit 68ad74d

Browse files
AndersonQmergify[bot]
authored andcommitted
libbeat: increase total_fields.limit to 12500 (#41640)
* libbeat: increase index template total_fields.limit to 12500 It increased the `index.mapping.total_fields.limit` from `10000` to `12500` in order to avoid ingestion failures caused by too many field in the index. Since 8.15.0 the limit started to be hit. The field count being exceeded is on the index, counting all mapped fields and the dynamic fields. That's why a small event might trigger the error, the event contains new fields to be mapped which would exceed the total field limit if mapped. (cherry picked from commit 42dd93b) # Conflicts: # CHANGELOG.asciidoc
1 parent f17e082 commit 68ad74d

3 files changed

Lines changed: 7 additions & 2 deletions

File tree

CHANGELOG.asciidoc

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -110,6 +110,7 @@ https://github.com/elastic/beats/compare/v8.15.3\...v8.15.4[View commits]
110110
*Affecting all Beats*
111111

112112
- Fix issue where old data could be saved in the memory queue after acknowledgment, increasing memory use. {pull}41356[41356]
113+
- Fix metrics not being ingested, due to "Limit of total fields [10000] has been exceeded while adding new fields [...]". The total fields limit has been increased to 12500. No significant performance impact on Elasticsearch is anticipated. {pull}41640[41640]
113114

114115
*Filebeat*
115116

@@ -212,6 +213,10 @@ https://github.com/elastic/beats/compare/v8.15.0\...v8.15.1[View commits]
212213

213214
*Affecting all Beats*
214215

216+
<<<<<<< HEAD
217+
=======
218+
- Beats Docker images do not log to stderr by default. The workaround is to pass the CLI flag `-e` or to set `logging.to_stderr: true` in the configuration file.
219+
>>>>>>> 42dd93b99b (libbeat: increase total_fields.limit to 12500 (#41640))
215220
- Beats stop publishing data after a network error unless restarted. Avoid upgrading to 8.15.1. Affected Beats log `Get \"https://${ELASTICSEARCH_HOST}:443\": context canceled` repeatedly. {issue}40705{40705}
216221

217222
==== Bugfixes

libbeat/template/load_test.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -170,7 +170,7 @@ func TestFileLoader_Load(t *testing.T) {
170170
"refresh_interval": "5s",
171171
"mapping": mapstr.M{
172172
"total_fields": mapstr.M{
173-
"limit": 10000,
173+
"limit": defaultTotalFieldsLimit,
174174
},
175175
},
176176
"query": mapstr.M{

libbeat/template/template.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ import (
3535
var (
3636
// Defaults used in the template
3737
defaultDateDetection = false
38-
defaultTotalFieldsLimit = 10000
38+
defaultTotalFieldsLimit = 12500
3939
defaultMaxDocvalueFieldsSearch = 200
4040

4141
defaultFields []string

0 commit comments

Comments
 (0)