NUTRINTG (6) File Sender (DEPRECATED/OUTDATED DOCUMENTATION)

File sender is responsible for sending files to the proper SFTP server based on received brandOrgCode. High level architecture overview:

Main process steps

File sender polls for messages from FileSenderQueue (process.fixed-delay=)
For each brandOrgCode from received message send process start uploading files from Inbound S3 bucket to the right SFTP server if the file is not currently sent (it is checked in SendFiles collection)
For each successful upload the information about success is stored in the SendFiles Mongo collection.
If all files listed in the message are uploaded successfully, the information is saved in ProcessedDelta collection, the message from the FileSenderQueue is removed and the email about successful processing is sent.

Spring integration channels graph and error handling

Below there is presented a channel graph of the file-sender:

Processing the message

The received message is split by brandOrgCode first.
The router routes the messages based on brandOrgCode or presence of the entity in SentFiles collection.

If the message has valid brandOrgCode it checks if the file is already sent to the right SFTP (SentFiles collection) If yes the message is sent to AlreadySendChannel If not it is redirected to the right upload gateway.
If the message has wrong brandOrgCode it redirects the message to NoBrandOrgCodeChannel for further processing.

All split messages are aggregated in

AggregatorChannel and sent as one aggregated result to result channel where the ProcessingDelta is saved, Inbound queue message is removed and successful email is sent.

Error handling

It is worth to notice that there are two more channels in the message channel graph:

General Error Channel - responsible for handling exceptions in the main flow, it sends email message with the details of the exception (ex. JSON format exception after receiving the message)
Aggregation Compensation Channel - handles all errors occurred in processing the one split message. The channel is responsible for sending the compensation message to the aggregation channel. It is required because all messages that were produced in the 1) Splitter have to be received all in aggregation channel to proceed the aggregation operation (ex. one upload failed). It is because split step produces messages which contains correlationId, sequenceNumber, sequenceSize headers and aggregator handler triggers only if all sequence messages are collected. The compensation channel gets the original message headers and adds it to the error message.

Configuration of new SFTP - BrandOrgCode mapping

Following keys have to be specified in application properties to define a new Upload SFTP gateway:

sftp.connections.<New BrandOrgCode>.user=YOUR_SFTP_USER
sftp.connections.<New BrandOrgCode>.private-key=keys/sfmc-ftp.ppk // YOUR RESOURCES PRIVATE KEY PATH
sftp.connections.<New BrandOrgCode>.remote-directory=Triggered_Automations/CDP_Import
sftp.connections.<New BrandOrgCode>.temporary-remote-directory=Triggered_Automations/CDP_Import/temp
sftp.connections.<New BrandOrgCode>.host=YOUR_SFTP_HOST
sftp.connections.<New BrandOrgCode>.port=22
sftp.connections.<New BrandOrgCode>.sessionTimeoutInMillis=30000
sftp.connections.<New BrandOrgCode>.sessionCacheSize=10

It is worth to mention that instead of <New BrandOrgCode> you have to put your brandOrgCode

Most important properties

Property	Description	Example

Property	Description	Example
`process.fixed-delay`	Polling interval in millis	10000
`process.delta-interval-in-days`	Interval in which we expect to receive full delta. It is required to preserve order of delta messages and it should be set to the same value as in file-set-analyzer service.	1
`process.parallelSendThreads`	The number of sending processes	4