AWS S3 Storage
This page describes how to connect an AWS S3 bucket to Solvimon for data exports. Solvimon can write to your bucket in two ways: by assuming an IAM role in your AWS account, or with the access key and secret of an IAM user.
Choose an authentication method
Use the IAM role unless you need Parquet files in the bucket.
IAM role (recommended)
With this method, you create an IAM role in your AWS account and allow Solvimon to assume it. For each export, Solvimon calls sts:AssumeRole with an external ID that only your platform uses. It then uploads the report file to your bucket with the temporary credentials it gets back. No access key is stored on either side.
Requirements
- An AWS account where you can create IAM roles and policies.
- An S3 bucket for the exports. To create one, follow the AWS documentation.
- To create the integration yourself instead of through Solvimon support: a Solvimon API key with permission to manage integrations and report subscriptions.
Setup
Get your external ID and the Solvimon principal from Solvimon
Contact Solvimon support to request an IAM role setup for S3 exports. Solvimon sends you two values:
Solvimon issues the external ID, and it’s unique to your platform. Don’t choose your own. A unique ID per customer is what stops another Solvimon customer from pointing an export at your role.
Create the permissions policy
In the IAM console, go to Policies → Create policy, open the JSON tab and paste the policy below. Replace {YOUR_BUCKET_NAME} with your bucket name and {YOUR_BUCKET_PATH} with the folder Solvimon writes to, for example solvimon.
Solvimon needs s3:ListBucket to check that it can reach the bucket (an S3 HeadBucket call) and s3:PutObject to upload each report. It doesn’t read, list or delete objects.
If the bucket uses SSE-KMS encryption with a customer managed key, add a statement that allows kms:GenerateDataKey on that key:
Create the role
In the IAM console, go to Roles → Create role and choose Custom trust policy. Paste the trust policy below, using the two values from Solvimon:
Attach the permissions policy from the previous step, name the role (for example solvimon-export) and create it. Copy the role ARN from the role’s summary page, for example arn:aws:iam::123456789012:role/solvimon-export.
Solvimon requests one-hour sessions, so keep the role’s maximum session duration at one hour or more. One hour is the AWS default.
Send the role details to Solvimon
Send these values to Solvimon support:
Solvimon creates the integration, checks that it can assume the role and reach the bucket, and points your report exports at it. To create the integration yourself, see Create the integration with the API.
Desk can only create S3 integrations that use an access key.
Create the integration with the API
Instead of sending the role details to Solvimon support, you can create the integration and the report exports yourself.
Create the integration
Create a DATA_EXPORT integration with POST /v1/integrations. Set authentication_type to IAM_ROLE and leave out the authentication block:
The response contains the integration’s id, which you need in the next steps.
Don’t send data_export.transport_type. Solvimon sets it from the authentication type and overwrites any value in the request.
Check the connection
After you create the integration, Solvimon assumes the role and checks that it can reach the bucket. Fetch the integration with GET /v1/integrations/{id} to see the result:
ACTIVE means the check passed. ERROR means it failed, and message contains the reason. The usual causes are a mismatch between the external ID in the trust policy and the integration, a wrong principal ARN in the trust policy, or a missing s3:ListBucket permission. Exports to an integration that is still PENDING or TESTING wait until the check finishes.
Send reports to the bucket
Point a report subscription at the integration with POST /v1/report-subscriptions. Set export_location.integration_id to the integration’s id:
See Reports for report configurations, schedules and parameters. To move an existing subscription to the new integration, update its export_location with PATCH /v1/report-subscriptions/{id}.
Where files are written
Solvimon builds each object key from three parts: the bucket path, the expanded path format, and the report’s file name.
- Bucket path:
export_location.s3.bucket_pathon the report subscription if set, otherwisebucket_pathon the integration. If you override it on the subscription, make sure the permissions policy covers that folder too. - Path format:
export_location.s3.path_formatis optional. It’s expanded in UTC at upload time and supports these placeholders:${YEAR},${MONTH},${DAY},${HOUR},${MINUTE},${SECOND},${MILLISECOND},${EPOCH}(milliseconds) and${UUID}. - Prefix or folder: if the last segment of the path format ends in
_or-, Solvimon uses it as a prefix on the file name. Otherwise the whole path format is a folder.
What you see in AWS
- File contents: Solvimon uploads the report file as it generated it, in one
PutObjectcall: a gzipped CSV file (.csv.gz).export_location.s3.format_typehas no effect with an IAM role, so files aren’t converted to Parquet. For Parquet files in your bucket, use the access key method. - Object metadata: each object has the user metadata
report-created-at,report-exported-at,source-last-modifiedandreport-resource-id(shown in S3 with thex-amz-meta-prefix). Usereport-resource-idto match a file to the report in Solvimon. - CloudTrail: each export appears as an
AssumeRoleevent for your role with the session namesolvimon-export-<platform ID>, followed by aPutObjectfrom that session.
Switch from an access key to an IAM role
You can’t change the authentication method of an existing S3 integration. To switch:
- Set up the role following the steps above and send the role details to Solvimon support.
- Solvimon creates a new integration and moves your report exports to it once it’s
ACTIVE. If you manage the exports through the API, update each report subscription soexport_location.integration_idpoints to the new integration. - When the first export arrives through the role, delete the old IAM user’s access key in AWS, then the user.
Access key and secret
Requirements
The following access is required for us to set-up a connection from Solvimon to your AWS S3 bucket.
- An AWS Account with IAM privileges to grant
readandlistpermissions for S3 buckets, or alternatively an existing S3 bucket with Access Key and Secret. - A project in AWS portal.
Setup
Create an AWS Access Key, Secret and a user policy
First of all, sign in to your AWS account. Follow the instructions to create an Access Key ID and Secret Access Key, which will be required to establish the connection to upload the Solvimon data to your S3 bucket. The IAM user associated with the credentials must be granted read and list permissions for the bucket and its objects. If you already have an existing Access Key and Secret, there is no need to create a new one.
This is a step to help you obtain the required permissions and credentials, and therefore the Access Key and Secret.
-
Log in to Amazon AWS and open the IAM console.
-
Go to
Policiesand then clickCreate Policies. -
Select
JSONtab, and paste the following JSON in the policy editor, by substituting the variables with your bucket name (replace{YOUR_BUCKET_NAME}). It is relevant to know that in this JSON there are not only the policies for the user, but also to be able to manage the S3 bucket properly.IAM policy -
Click now
Create policy. -
In the IAM dashboard, go to
Usersand either create a new one or use an existing one. -
Go to
Add Permissionsand selectAttach policies directly, where you have to find the policy that has just been created. -
Once the user with its permissions is created, select the
Security credentialstab and clickCreate access key. Click Create access key to generate the keys.
Create an S3 Bucket
The S3 Bucket will be the destination where the data coming from Solvimon will be exported. To create a bucket, in case you don’t have one already for this purpose, you can follow this documentation.