AWS S3 - Fluent Bit
Description
The Amazon S3 output plugin allows to ingest your records into the S3 service. See fluent bit page for more information.
Parameters
| Parameter | Description | Type | Default |
|---|---|---|---|
| region | The AWS region of your S3 bucket. | str | "us-east-1" |
| bucket | S3 Bucket name. | str | N/A (Mandatory) |
| json_date_key | Specify the name of the time key in the output record. To disable the time key just set the value to false. | str | "date" |
| json_date_format | Specify the format of the date. Supported formats are double, epoch, iso8601 (eg: 2018-05-30T09:39:52.000681Z) and java_sql_timestamp (eg: 2018-05-30 09:39:52.000681). | str | "iso8601" |
| total_file_size | Specifies the size of files in S3. Maximum size is 50G, minimim is 1M. | str | "100M" |
| upload_chunk_size | The size of each 'part' for multipart uploads. Max: 50M. | str | "50M" |
| upload_timeout | Whenever this amount of time has elapsed, Fluent Bit will complete an upload and create a new file in S3. For example, set this value to 60m and you will get a new file every hour. | str | "10m" |
| store_dir | Directory to locally buffer data before sending. When multipart uploads are used, data will only be buffered until the upload_chunk_size is reached. S3 will also store metadata about in progress multipart uploads in this directory; this allows pending uploads to be completed even if Fluent Bit stops and restarts. It will also store the current $INDEX value if enabled in the S3 key format so that the $INDEX can keep incrementing from its previous value after Fluent Bit restarts. | str | "/tmp/fluent-bit/s3" |
| store_dir_limit_size | The size of the limitation for disk usage in S3. Limit the amount of s3 buffers in the store_dir to limit disk usage. Note: Use store_dir_limit_size instead of storage.total_limit_size which can be used to other plugins, because S3 has its own buffering system. | int | 0 |
| s3_key_format | Format string for keys in S3. This option supports a UUID, strftime time formatters, a syntax for selecting parts of the Fluent log tag using a syntax inspired by the rewrite_tag filter. Add $UUID in the format string to insert a random string. Add $INDEX in the format string to insert an integer that increments each upload. The $INDEX value will be saved in the store_dir so that if Fluent Bit restarts the value will keep incrementing from the previous run. Add $TAG in the format string to insert the full log tag; add $TAG[0] to insert the first part of the tag in the s3 key. The tag is split into “parts” using the characters specified with the s3_key_format_tag_delimiters option. Add extension directly after the last piece of the format string to insert a key suffix. If you want to specify a key suffix and you are in use_put_object mode, you must specify $UUID as well. More explanations can be found in the S3 Key Format explainer section further down in this document. See the in depth examples and tutorial in the documentation. Time in s3_key is the timestamp of the first record in the S3 file. | str | "/fluent-bit-logs/$TAG/%Y/%m/%d/%H/%M/%S" |
| static_file_path | Disables behavior where UUID string is automatically appended to end of S3 key name when $UUID is not provided in s3_key_format. $UUID, time formatters, $TAG, and other dynamic key formatters all work as expected while this feature is set to true. | bool | false |
| s3_key_format_tag_delimiters | A series of characters which will be used to split the tag into 'parts' for use with the s3_key_format option. See the in depth examples and tutorial in the documentation. | str | N/A |
| use_put_object | Use the S3 PutObject API, instead of the multipart upload API. When this option is on, key extension is only available when $UUID is specified in s3_key_format. If $UUID is not included, a random string will be appended at the end of the format string and the key extension cannot be customized in this case. | bool | false |
| role_arn | ARN of an IAM role to assume (ex. for cross account access). | str | N/A |
| endpoint | Custom endpoint for the S3 API. An endpoint can contain scheme and port. | str | N/A |
| sts_endpoint | Custom endpoint for the STS API. | str | N/A |
| canned_acl | Predefined Canned ACL policy. | str | N/A |
| compression | Compression type for S3 objects. 'gzip' is currently the only supported value by default. If Apache Arrow support was enabled at compile time, you can also use 'arrow'. For gzip compression, the Content-Encoding HTTP Header will be set to 'gzip'. Gzip compression can be enabled when use_put_object is 'on' or 'off' (PutObject and Multipart). Arrow compression can only be enabled with use_put_object On. | str | N/A |
| content_type | A standard MIME type for the S3 object; this will be set as the Content-Type HTTP header. | str | N/A |
| send_content_md5 | Send the Content-MD5 header with PutObject and UploadPart requests, as is required when Object Lock is enabled. | bool | false |
| auto_retry_requests | Immediately retry failed requests to AWS services once. This option does not affect the normal Fluent Bit retry mechanism with backoff. Instead, it enables an immediate retry with no delay for networking errors, which may help improve throughput when there are transient/random networking issues. | bool | true |
| log_key | By default, the whole log record will be sent to S3. If you specify a key name with this option, then only the value of that key will be sent to S3. For example, if you are using Docker, you can specify log_key log and only the log message will be sent to S3. | str | N/A |
| preserve_data_ordering | Normally, when an upload request fails, there is a high chance for the last received chunk to be swapped with a later chunk, resulting in data shuffling. This feature prevents this shuffling by using a queue logic for uploads. | bool | true |
| storage_class | Specify the storage class for S3 objects. If this option is not specified, objects will be stored with the default 'STANDARD' storage class. | str | N/A |
| retry_limit | Integer value to set the maximum number of retries allowed. For previous version, the number of retries is 5 and is not configurable. | int | 1 |
| external_id | Specify an external ID for the STS API, can be used with the role_arn parameter if your role requires an external ID. | str | N/A |
Node configuration
...
flb_s3:
plugin: "dc_destinations/FlbS3"
inputs: ["/dc/group/data"]
bucket: my-bucket
region: us-west-2
total_file_size: "50M"
use_put_object: false
compression: "gzip"
s3_key_format: "/$TAG/%Y/%m/%d/%H_%M_%S.gz"
...