NUTRINTG Shopify to CDP Integration
- Sebastian Baltruszewicz
Preconditions:
Brand-Org-Code is unique globally (particular Brand-Org-Code will not be present within two different Markets / Countries).
Order data
Shopify → Middleware → CDP
Epsilon CDP spec: RB_E-Commerce_Functional_Specification V_1.8.docx
Steps
SERVICE EP-275 - Getting issue details... STATUS
- Receive Shopify webhook type: orders/create - HTTP request with JSON payload.
RB endpoint should have valid SSL certificate, since Shopify verifies SSL certificates when delivering payloads to HTTPS webhook addresses. - Acknowledge receiving data by sending 200 OK responds within 5 seconds. If there is no response, or an error is returned, then Shopify retries the connection 19 times over the next 48 hours. A webhook is deleted if there are 19 consecutive failures. Any response outside of the 200 range, including 3XX HTTP redirection codes, indicates that you did not receive the webhook.
- Validate json payload:
- whether X-Shopify-Topic header field contains "orders/create" or "order/updated"value
- whether X-Shopify-Shop-Domain header field is in the integrated stores list on MongoDB (source: Shopify URL Mapping.xlsx)
- whether X-Shopify-Hmac-SHA256 header field against correct HMAC (list was shared with Tristan) EP-230 - Shopify Error handling and webhook verification ON HOLD
- IF any of the validations returned false → store json to the Error queue and return HTTP code 400.
ELSE → continue flow. - Change customer-id field to: BrandOrgCode (from Shopify URL Mapping.xlsx) & "SHPF" & customer_id. Example: FRADURSHPF207119551
- Change value of "shipping_address-country" and "billing_address-country_code" with value from "Market code" column from in the Shopify URL Mapping.xlsx (URL Mapping tab).
- Change "created_at" field:
name: to "order_date"
value: adjust to format YYYY-MM-DDTHH:MM:SS and UTC timezone - Add to json fields from "Additional fields table" below
- List of required fields and order (different from original json) is provided in the Shopify URL Mapping.xlsx
- Sends the record to the Middleware Router queue. Router decide if the message should go to cdp-shopify-order-adapter or cds-middleware-message-adapter
- IF any of the SERVICE steps failed return HTTP code 500 and send email - titled "Shopify Order Service Flow Failure" with order_id and exception description in the content.
EP-348 - Getting issue details... STATUS
EP-351 - Getting issue details... STATUS
EP-350 - Getting issue details... STATUScdp-shopify-order-adapter
- Read messages from the Shopify Order queue
- Save record as CSV(.txt) file at EC2 catalogue (one order / json = one file):
<yyy_mm_dd_hh> catalogue in UTC timezone
→ "<X-Shopify-Shop-Domain>" catalogue (field in the json header, example: "https://durexfrance.myshopify.com") - Create file for each "X-Shopify-Shop-Domain" containing records from last 24h, by merging appropriate files with bash script. The process executes once per day at a specific hour (configurable) and contains all records from that hour the day before up to the time of execution (e.g.: if that hour is specified at 3 p.m. and today is 12 March, then all records from 11 March 3:00:00 p.m. inclusive to 12 March 3:00:00 p.m. exclusive).
- Name files according to File naming table (v1).
- Create an empty file with headers for each store without orders.
- Add headers according to Shopify URL Mapping.xlsx
- Convert to CSV (.txt)
PGP encrypt files. Production key: frms_prod_key_pub.asc . Tests key: public_key-narnia.epsilon.com.asc
Create MD5
Transfer files to incoming folder of the sFTP locations for the respective country (market code) before 19:30 UTC every day (the uploading should be finished by this time.)
The folder names will be the same as the 3-character ISO country codes ("Market code" column in the Shopify URL Mapping.xlsx) for the respective countries.
Path: Incoming/Shopify/<3 letter ISO Country_Name>. sFTP credentials are provided in the table below.- Move transferred (encrypted) files to "Archive" folder for backup purposes. For this folder data retentions is set for 21 days. EP-296 - Getting issue details... STATUS
- IF any of the ADAPTER steps failed email is sent - titled "Shopify Order Adapter Flow Failure" with folder name and exception description in the content.
cds-middleware-message-adapter
- Read message from CDS Adapter Queue
- Check configuration for AccountSource code.
- If the confgiuration does not exists then drops the message (IMPORTANT! in given situation the client does not know the request has been dropped since it is an async communication)
- Otherwise CDS Adapter retrieve configuration from the database (bucketId, url, brand, market) for given ASC
- Sends request to CDS.
- In case of any failure: the same as indicates the point no 5.
# in the final file | Field name | Source (column in the Shopify URL Mapping.xlsx) |
---|---|---|
45 | brand_org_code | Brand-Org-Code |
46 | language_code | Language code |
47 | vendor | vendor |
48 | txn_src_code | txn_src_code |
49 | acct_src_code_cust | Account_Source_Code Customer data |
50 | acct_src_code_ship | Account_Source_Code Shipping data |
Behavior when sent header X-Shopify-Topic="orders/updated"
In case of request is sent to CDS - the new version of record is created. What is important the email address must not change, otherwise, two separate records will be created. The new version of the records cuases, that when one consumes search API to CDS, the latest version of row is returned. In case if there is an integration with SFMC, it depends on the DE settings, proper primary key must be set. For this topic please reach out to SF Team.
In case of request is sent to CDP - the most important, orderId must be the same, which is pretty obvious. Middleware sends orders to CDP in batches every 24 hrs. If the order is created and then updated in the same period of 24 hrs, then only the updated order is sent to CDP. Otherwise, one day the created order data is sent to CDP, and another day its updated version is sent to CDP. Then it is CDP reposnbility to handle the situation properly. For this scenario, the requirements are given below (Epsilon statement regards order updates)
In the Order, the number of line items should not be increased or decreased. In fact, there should be no change in the line items, otherwise, data corruption might happen. The FS document has the following assumptions. AS009 We will not be receiving any updates for orders that change the number of lines since we receive only the final order. AS011 Epsilon will accept all the financial status values and update the transaction if the financial status change. Currently No logic exists to delete items or reorder item numbers if the content of the items changes between versions. RB will not make changes to the number of items in the order.
File naming table
X-Shopify-Shop-Domain | Filename | Filename v2 | Filename v3 |
---|---|---|---|
https://durexfrance.myshopify.com | RB_FR_ORDER_RBFRADUR_SHOPIFY_<timestamp>.json | ->.txt | ->.txt.pgp |
https://durexgermany.myshopify.com | RB_DE_ORDER_RBDEUDUR_SHOPIFY_<timestamp>.json | ->.txt | ->.txt.pgp |
https://durexspain.myshopify.com | RB_ES_ORDER_RBESPDUR_SHOPIFY_<timestamp>.json | ->.txt | ->.txt.pgp |
https://durexuk.myshopify.com | RB_EN_ORDER_RBGBRDUR_SHOPIFY_<timestamp>.json | ->.txt | ->.txt.pgp |
https://enfashop-es.myshopify.com | RB_ES_ORDER_MJNESP_SHOPIFY_<timestamp>.json | ->.txt | ->.txt.pgp |
https://enfashop-pl.myshopify.com | RB_PL_ORDER_MJNPOL_SHOPIFY_<timestamp>.json | ->.txt | ->.txt.pgp |
https://schollfrance.myshopify.com | RB_FR_ORDER_RBFRASCO_SHOPIFY_<timestamp>.json | ->.txt | ->.txt.pgp |
https://schollgermany.myshopify.com | RB_DE_ORDER_RBDEUSCO_SHOPIFY_<timestamp>.json | ->.txt | ->.txt.pgp |
https://scholluk.myshopify.com | RB_EN_ORDER_RBGBRSCO_SHOPIFY_<timestamp>.json | ->.txt | ->.txt.pgp |
https://durex-uk-test-store.myshopify.com | RB_EN_ORDER_RBGBRDUR_SHOPIFY_<timestamp>.json | ->.txt | ->txt.pgp |
<timestamp> = File creatation datetime. Only digits in UTC timezone, example: "20191207040627".
Examplary filename: RB_EN_ORDER_RBGBRDURX_SHOPIFY_20191207040627.txt.pgp
File specification | |
---|---|
Format | CSV (.txt) |
Delimiter | comma, Additionally all values should be enclosed by double quotes. |
Terminator | CRLF |
Headers | required (names according to Shopify URL Mapping.xlsx) The case of the headers does not matter. It can be sent as “eventtype” or “EVENTTYPE”. |
sFTP credentials
HOST | |
---|---|
Port | 22 |
UAT Username | rbnk_q_ecom |
PROD Username | rbnk_p_ecom |
EP-333 - Getting issue details... STATUS
Additional documentation
Shopify API: https://help.shopify.com/en/api/guides
Shopify webhooks API: https://help.shopify.com/en/api/reference/events/webhook
Mule ObjectStore: https://docs.mulesoft.com/object-store/
Customer data
Shopify → Middleware → CDP
Steps
SERVICE
EP-280
-
Getting issue details...
STATUS
- REST RB endpoint receives Shopify webhook (types: customers/create, customers/update)
- Validate json payload:
- whether X-Shopify-Topic header field contains " customers/create" or "customers/update" values
- whether X-Shopify-Shop-Domain header field is in the integrated stores list on MongoDB (source: Shopify URL Mapping.xlsx)
- whether X-Shopify-Hmac-SHA256 header field against correct HMAC (list was shared with Tristan) EP-230 - Shopify Error handling and webhook verification ON HOLD
whether JSON schema is correct. shopify-customer-schema.json
- IF payload didn't pass validation move it to Error queue (secured by another queue - Dead letter queue) and return HTTP code 400.
ELSE → continue flow. - Add LDS agreements data to the json body according to Body table below.
- Store data as json in the queue (Anypoint MQ)
- IF processing payload failed return HTTP code 500.
ADAPTER EP-273 - Getting issue details... STATUS - Read queue
- Create json header and body according to below table including
- IF webhook type = customers/update
- Send json to Epsilon's API secured by oauth2:
- Add Profile API Post call - page 47 of MJN_Epsilon Integration API Document v1.36.pdf IF webhook type = customers/create
- Update Profile by ID API Post call - page 93 of MJN_Epsilon Integration API Document v1.36.pdf IF webhook type = customers/update. Adding ProfileId as URL parameter - api/v1/profiles/{profileId}.
Get ProfileId by "Get Profiles (Search)" CDP API method - page 162, providing SourceAccountNumber according to Body table below.
- CDP responds with Http 201 code in the header and sent data in the body
SERVICE to ADAPTER data:
FIELD | TYPE | VALUE | |
---|---|---|---|
operation | string | "CREATE" - if webhook type = customers/create "UPDATE" - if webhook type = customers/update | |
acceptLanguage | string | "en-US" | |
programCode | string | → "Brand-Org-Code" column in the ("URL Mapping" tab) Shopify URL Mapping.xlsx | |
brandOrgCode | string | → "Brand-Org-Code" column in the ("URL Mapping" tab) Shopify URL Mapping.xlsx | |
accountSource | string | → "Account_Source_Code Customer data" column in the "URL Mapping" tab) Shopify URL Mapping.xlsx | |
sourceAccountNumber | string | → "sourceAccountNumber" column in the ("URL Mapping" tab) Shopify URL Mapping.xlsx | |
profileData | object | → "Customer" tab Shopify URL Mapping.xlsx If "addresses → phone" value in the Shopify webhook is NULL or empty, then send empty "Phones" array. Since PhoneNumber" value is required for each instance of the Phones. EP-427 - Getting issue details... STATUS |
ADAPTER to CDP API data:
Body = profileData object content (as above)
Header json fields sent by ADAPTER to CDP API:
Header field | Value |
---|---|
Authorization | Token - should be refreshed every 48h |
Accept-Language | Provided by SERVICE |
Program-Code | Provided by SERVICE |
Brand-Org-Code | Provided by SERVICE |
Account-source | Provided by SERVICE |
Source-Account-Number | Provided by SERVICE ??? |
Content-Type | Token Calls - application/x-www-form- urlencoded |
External-Correlation-Id | UID |
UPDATING profile
EP-369 - Getting issue details... STATUS
In order to update customer profile in CDP request to "Update Profile by ID" of CDP API is sent. Method is described on page 93 of MJN_Epsilon Integration API Document v1.36.pdf
Below there are additional requirements for the content of the updated objects in the request comparing to profile creating.
Object to update | "Update Profile by ID" CDP API method (page 93) |
---|---|
Profile |
|
Addresses | Provide the same "AddressId" value - retrieve it from CDP, take one where "SourceAccountNumber" match. |
Emails | Provide the same "EmailId" value - retrieve it from CDP, take one where "SourceAccountNumber" match. |
Phones | Provide the same "PhoneId" value - retrieve it from CDP, take one where "SourceAccountNumber" match. |
SocialAccounts | Provide the same "ProfileSocialAccountId" value - retrieve it from CDP, take one where "SourceAccountNumber" match. |
Children | Provide the same "ProfileChildId" value - retrieve it from CDP, take one where "BirthDate" (if still there are two objects than + "FirstName" and "LastName") match. *Because FirstName and LastName are not obligatory. |
ProfileSubscriptions | Provide the same "SubscriptionId" and "ChannelCode" values. |
Agreeements | Provide the same "BusinessId" value - take it from "LDS agreements" tab Shopify URL Mapping.xlsx |
UnmappedAttributes | Is overwritten each time with any provided value. |
ProfileItemList | Provide the same key (property) name. |
Abandoned checkout directly to SFMC
I as Middleware want to obtain checkouts data from Shopify by API and send abandoned ones data directly to SFMC, so that SFMC is able to send the email reminder to the customer even 1 hour after the abandonment. Since Shopify provides API only for all checkouts, I as Middleware want to consider as abandoned, checkouts which haven't been finished within hour after last update.
Since reminder email contains images, CDP Middleware requests also product images links and then combines data from both sources to one request sent to SMFC API.
Tasks in Jira:
EP-309
-
Getting issue details...
STATUS
EP-334
-
Getting issue details...
STATUS
API Leak rate limit: 4 requests per second
REQUEST AUTHORISATION
- Basic HTTP authentication by using their Admin API key and password as a username and password OR
- by including to the request header
X-Shopify-Access-Token: {access_token}
, where{access_token}
is replaced by this store Admin API password.
More details are available in the Shopify API Authorisation Documentation
Get abandoned checkout data (getAbandonedCheckout)
ENDPOINT
endpoint = {storeDomain}/admin/api/2019-07/checkouts.json?limit=250&updated_at_min={ startDate }&updated_at_max={ endDate }
method = GET
- storeDomain = domain of the store (ended with myshopify.com) = X-Shopify-Shop-Domain column in the "URL Mapping" tab of Shopify URL Mapping.xlsx
Each store in the scope (available in the "URL Mapping" tab) is requested - "2019-07" - version of the Shopify API
- "limit=250" - number of returned by Shopify API checkouts
- "updated_at_min=..." - query to restrain results to those updated at after the provided date.
- "updated_at_max=..." - query to restrain results to those updated at before the provided date.
- startDate = now - 65 minutes. Value in ISO 8601 format, example value: 2019-07-10T00:15:42+00:00.
- endDate = now - 60 minutes. Value in ISO 8601 format, example value: 2019-07-10T00:15:47+00:00.
HOW OFTEN request is send to each store
Each 5 min.
EXPECTED ANSWER
- HTTP/1.1 200 OK
- Json with up to 250 checkouts, not older than 3 months and ordered from the oldest to the newest one
{ "checkouts": [ { "id": 26371164, "token": "zs9ru89kuqcdagk8bz4r9hnxt22wwd42", "cart_token": "68778783ad298f1c80c3bafcddeea02f", "email": "bob.norman@hostmail.com", "gateway": "bogus", "buyer_accepts_marketing": false, "created_at": "2012-10-12T07:05:27-04:00", "updated_at": "2012-10-12T07:05:27-04:00", "landing_site": null, "note": null, "note_attributes": [ { "name": "custom engraving", "value": "Happy Birthday" }, { "name": "colour", "value": "green" } ], "referring_site": null, "shipping_lines": [ { "title": "Free Shipping", "price": "0.00", "code": "Free Shipping", "source": "shopify", "applied_discounts": [], "id": "5da41c1738454765" } ], "taxes_included": false, "total_weight": 400, "currency": "USD", "completed_at": null, "closed_at": null, "user_id": null, "location_id": null, "source_identifier": null, "source_url": null, "device_id": null, "phone": null, "customer_locale": null, "line_items": [ { "applied_discounts": [], "key": 49148385, "destination_location_id": null, "fulfillment_service": "manual", "gift_card": false, "grams": 200, "origin_location_id": null, "product_id": 632910392, "properties": null, "quantity": 10, "requires_shipping": true, "sku": "IPOD2008RED", "tax_lines": [], "taxable": true, "title": "IPod Nano - 8GB", "variant_id": 49148385, "variant_title": "Red", "variant_price": null, "vendor": "Apple", "user_id": null, "unit_price_measurement": null, "country_hs_codes": [], "country_code_of_origin": null, "province_code_of_origin": null, "harmonized_system_code": null, "compare_at_price": null, "line_price": "1990.00", "price": "199.00" }, { "applied_discounts": [], "key": 808950810, "destination_location_id": null, "fulfillment_service": "manual", "gift_card": false, "grams": 200, "origin_location_id": null, "product_id": 632910392, "properties": null, "quantity": 10, "requires_shipping": true, "sku": "IPOD2008PINK", "tax_lines": [], "taxable": true, "title": "IPod Nano - 8GB", "variant_id": 808950810, "variant_title": "Pink", "variant_price": null, "vendor": "Apple", "user_id": null, "unit_price_measurement": null, "country_hs_codes": [], "country_code_of_origin": null, "province_code_of_origin": null, "harmonized_system_code": null, "compare_at_price": null, "line_price": "1990.00", "price": "199.00" } ], "name": "#26371164", "source": null, "abandoned_checkout_url": "https://checkout.local/690933842/checkouts/zs9ru89kuqcdagk8bz4r9hnxt22wwd42/recover", "discount_codes": [], "tax_lines": [ { "price": "11.94", "rate": 0.06, "title": "State Tax" } ], "source_name": "web", "presentment_currency": "USD", "total_discounts": "0.00", "total_line_items_price": "3980.00", "total_price": "3991.94", "total_tax": "11.94", "subtotal_price": "3980.00", "billing_address": { "first_name": "Bob", "address1": "Chestnut Street 92", "phone": "555-625-1199", "city": "Louisville", "zip": "40202", "province": "Kentucky", "country": "United States", "last_name": "Norman", "address2": "", "company": null, "latitude": 45.41634, "longitude": -75.6868, "name": "Bob Norman", "country_code": "US", "province_code": "KY" }, "shipping_address": { "first_name": "Bob", "address1": "Chestnut Street 92", "phone": "555-625-1199", "city": "Louisville", "zip": "40202", "province": "Kentucky", "country": "United States", "last_name": "Norman", "address2": "", "company": null, "latitude": 45.41634, "longitude": -75.6868, "name": "Bob Norman", "country_code": "US", "province_code": "KY" }, "customer": { "id": 207119551, "email": "bob.norman@hostmail.com", "accepts_marketing": false, "created_at": "2019-06-28T15:06:47-04:00", "updated_at": "2019-06-28T15:06:47-04:00", "first_name": "Bob", "last_name": "Norman", "orders_count": 1, "state": "disabled", "total_spent": "199.65", "last_order_id": 450789469, "note": null, "verified_email": true, "multipass_identifier": null, "tax_exempt": false, "phone": null, "tags": "", "last_order_name": "#1001", "currency": "USD", "accepts_marketing_updated_at": "2005-06-12T11:57:11-04:00", "marketing_opt_in_level": null, "admin_graphql_api_id": "gid://shopify/Customer/207119551", "default_address": { "id": 207119551, "customer_id": 207119551, "first_name": null, "last_name": null, "company": null, "address1": "Chestnut Street 92", "address2": "", "city": "Louisville", "province": "Kentucky", "country": "United States", "zip": "40202", "phone": "555-625-1199", "name": "", "province_code": "KY", "country_code": "US", "country_name": "United States", "default": true } } } ] }
EXCEPTION
IF Shopify returned 250 checkouts THEN
1. Count number of checkouts for this period of time (5 min), sending request to endpoint:
endpoint = {storeDomain}/admin/api/2019-07/checkouts/count.json?updated_at_min={ startDate }&updated_at_max={ endDate }
method = GET
- exception.StartDate = (datetime of the request which received 250 checkouts) - 65 minutes. Value in ISO 8601 format, example value: 2019-07-10T00:15:42+00:00.
- exception.EndDate.Exception = (datetime of the request which received 250 checkouts) datetime now - 60 minutes. Value in ISO 8601 format, example value: 2019-07-10T00:15:47+00:00.
// 20190807130008 // https://durexuk.myshopify.com/admin/api/2019-04/checkouts/count.json?updated_at_min=2019-07-22T00:09:47-01:00&updated_at_max=2019-07-22T09:18:52-01:00 { "count": 32 }
2. Repeat requests for "count" times, until all checkouts for these five minutes are retrieved.
3. Go back to normal process
HOW checkout data is STORED
Abandoned checkouts (where "completed_at": null) data is saved to AnypointMQ, where it waits for consolidation with product images data.
Get product images data (getProductImage)
As Middleware I want to have a database of product Images links of every integrated Shopify store, so that SFMC receives full abandoned checkout data and able to send email reminders containing product images.
ENDPOINT
endpoint = {storeDomain}/admin/api/2019-07/products.json?limit=250&fields=image
method = GET
- storeDomain = domain of the store (ended with myshopify.com) = X-Shopify-Shop-Domain column in the "URL Mapping" tab of Shopify URL Mapping.xlsx
Each store in the scope (available in the "URL Mapping" tab) is requested. - "2019-07" - version of the Shopify API
- "limit=250" - number of returned by Shopify API checkouts. 250 is above max number of products one RB store is offering - Scholl UK 201 product, but it can increase once a year, so it should be monitored.
- "fields=image" - constraining response to only image related fields
// 20190726155851 // https://durex-uk-test-store.myshopify.com/admin/api/2019-04/products.json?fields=image { "products": [ { "image": { "id": 5430477062202, "product_id": 628184809530, "position": 1, "created_at": "2018-08-22T13:36:51+01:00", "updated_at": "2018-08-22T13:36:51+01:00", "alt": "7 Nights Of Fun", "width": 1024, "height": 1024, "src": "https://cdn.shopify.com/s/files/1/0070/7513/5546/products/7-nights-of-fun-col-condoms-lubes-pleasure-sets-sex-toys-relationship-advice-durex-uk-test-store_674.jpg?v=1534941411", "variant_ids": [ ] } }, { "image": { "id": 6569550577722, "product_id": 2120385560634, "position": 1, "created_at": "2018-11-09T14:34:17+00:00", "updated_at": "2018-11-09T14:34:17+00:00", "alt": "Carry On Travelling, front", "width": 1500, "height": 1500, "src": "https://cdn.shopify.com/s/files/1/0070/7513/5546/products/carry-on-travelling-bundle-front-durex-uk_49fbca5b-eeff-49f3-a792-2b814c77408d.jpg?v=1541774057", "variant_ids": [ ] } } ] }
HOW OFTEN image request is sent to each store
Every 4 hours
HOW images are STORED and UPDATED
IF any of the objects has "updated_at" field with value > lastProductImageRequestDate THEN productImagedatabase for this specific store is overwrited.
Id est below values from "products" [ "image": {} ] object overwrite existing values.
- "product_id" field value in the "productID" column / field.
- "id" field value in the "imageId" column / field
- "src" field value in the "productImage" column / field.
productImagedatabase = DynamoDB. Database contain images for all integrated stores, since product_id is unique across entire Shopify platform. Productimage can be accessed by dedicated function with productID as input.
Get Individual_ID (getIndividualId)
EP-463 - Getting issue details... STATUS
- IndividualId value is retrieved from CDP by "Get Profiles (Search)" CDP API method - page 162 of MJN_Epsilon Integration API Document v1.36.pdf , providing appropriate "SourceAccountNumber".
SourceAccountNumber is constructed in Middleware in the below way:
BrandOrcCode & “_” & customer_id from Shopify API received within 374145073. Path in Json: checkouts → customer → id
- If no account is found, the data has to be sent to Salesforce with other event ID and with email as ContactKey.
Consolidate checkout and product images data (ConsolidateData)
As Middleware I want to combine abandoned checkout and product images data, so that I am able to send a request to SFCM containing the required set of information.
Consolidation and transfer to SFMC is happening realtime, as soon as there is abandoned checkout data, it is enriched with "productImage" image field, Individual_ID and send to SFMC.
- Abandoned checkout data is taken form AnypointMQ
- product image is taken from productImagedatabase
- Individual_ID is taken from 374145073
- "productImage" field of this store productImagedatabase WHERE productImagedatabase.productID = shopifywebhook.line_items.product_Id is added it to checkout data.
- Individual_ID is added
Mapping of the fields sent to SFMC (available also in the "Abandoned checkout" tab of Shopify URL Mapping.xlsx):
# | Shopify / productImagedatabase Field name | File and SFMC field Name | Field Data Type | Max Length | Req (Y/N) | Description/Notes |
---|---|---|---|---|---|---|
1 | id | checkout-id | long | Y | Id of the checkout. Should be added on CDP side, before sending to CDP as batchfile | |
2 | individual_id | individual_id | string | 255 | Y | Individual_ID - ID of the person in CDP. Should be added on CDP side, before sending to CDP as batchfile |
3 | customer -> first_name | checkout-customer-first_name | string | 60 | Y | First Name – required |
4 | customer -> last_name | checkout-customer-last_name | string | 60 | Y | Last Name – required |
5 | abandoned_checkout_url | abandoned_checkout_url | string | 1024 | Y | The recovery URL to recover the cart |
6 | buyer_accepts_marketing | buyer_accepts_marketing | boolean | Y | Indicates whether the person accepts marketing (TRUE=Yes; False=No) | |
7 | cart_token | cart_token | string | 32 | Y | The unique ID for the cart |
8 | created_at | created_at | string | 19 | Y | The date/time when the order was created (YYYY-MM-DDTHH:MM:SS) |
9 | string | 254 | Y | The customer's email address | ||
10 | referring_site | referring_site | string | 500 | N | The website that the customer clicked on to come to the shop |
11 | subtotal_price | subtotal_price | string | 38 | Y | Price of the order before shipping and taxes |
12 | taxes_included | taxes_included | string | 10 | Y | Indicates whether taxes are included ( (TRUE=Yes; False=No) |
13 | total_discounts | total_discounts | string | 38 | Y | The total of the discounts applied to the order |
14 | total_line_items_price | total_line_items_price | string | 38 | Y | The sum of the prices of all line items in the checkout in presentment currency. |
15 | total_price | total_price | string | 38 | Y | The sum of all prices of items in the order including taxes and discounts |
16 | total_tax | total_tax | string | 38 | Y | The sum of all the taxes applied to the checkout in presentment currency. |
17 | total_weight | total_weight | number | Y | The sum of the weights of all items in the order | |
18 | updated_at | updated_at | string | 19 | Y | The date/time when the checkout was last modified (YYYY-MM-DDTHH:MM:SS) format |
19 | name | checkout_name | string | 255 | N | The checkout name as represented by a number. E.g. #981820079255243537 |
20 | token | checkout_token | string | 255 | N | A unique ID for a checkout. E.g. b1946ac92492d2347c6235b4d2611184 |
21 | shipping_address -> first_name | shipping_address-first_name | string | 60 | N | Header level. First name associated to the Shipping address |
22 | shipping_address -> last_name | shipping_address-last_name | string | 60 | N | Header level. Last name associated to the Shipping address |
23 | shipping_lines -> title | shipping_lines-title | string | 40 | N | Header level. The title of the shipping method, e.g. Small Packet International Air. |
24 | discount_codes -> discount_code -> code | discount_code | string | 20 | N | Header level. Discount code. Only one code will be available in the "discount_codes" object. |
25 | {counting line items starting from 1} | line_items-line_number | integer | N | Details level. Position of the line item in the order | |
26 | line_items -> sku | line_items-sku | string | 30 | N | Details level. Product SKU |
27 | line_items -> title | line_items-title | string | 1000 | N | Details level. Product name |
28 | line_items -> price | line_items-price | string | 38 | N | Details level. Product price |
29 | line_items -> quantity | line_items-quantity | integer | N | Details level. Quantity purchased | |
30 | "productImage" field of this store productImagedatabase, WHERE productImagedatabase.productID = shopifywebhook.line_items.product_Id | line-items-image_url | string | 1024 | N | Details level. URL to item image |
Json sent to SMFC
{ "checkout-id": 7877338071086, "checkout-customer-first_name": "Tomasz", "checkout-customer-last_name":"Lem", "abandoned_checkout_url": "https://durex-uk-husky-test.myshopify.com/3456172078/checkouts/1d4505800ad62fb354a225f3b1dcc22c/recover?key=a2253df3135f9a18df6307d3ca7d9d5d", "buyer_accepts_marketing": true, "cart_token": "17dc8320f2d9f0fb49ca94ce11f79180", "created_at": "2019-06-13T08:10:33-04:00", "email": "tomasz.lem@mail.net", "referring_site": "", "subtotal_price": "170.39", "taxes_included": true, "total_discounts": "0.00", "total_line_items_price": "170.39", "total_price": "180.39", "total_tax": "31.86", "total_weight": 968, "updated_at": "2019-08-12T06:03:57-04:00", "checkout_name": "#7877338071086", "checkout_token": "1d4505800ad62fb354a225f3b1dcc22c", "shipping_address-first_name": "Tomasz", "shipping_address-last_name": "Lem", "shipping_lines-title": "Small Packet International Air", "discount_code": "QDFKWEJ", "line_items":[ { "line_items-line_number": 1, "line_items-sku": "3030446", "line_items-title": "Durex Thin Feel Condoms 30 Pack", "line_items-price": "15.49", "line_items-quantity": 11, "line-items-image_url": "https://cdn.shopify.com/s/files/1/2992/1628/products/durex-uk-condoms-durex-surprise-me-condoms-40-pack-7178711957586_540x.jpg?v=1561979155" } ], "individual_id": "4020008919656" }
IF productImage" field of this store productImagedatabase is not available, getProductImage method is forced and combining for this product is repeated.
Send to SFMC (sendToSFMC)
Data is sent to SFMC Journey Builder API.
Appendix Item 1 - Journey Builder API
The API Entry Event gives you two things:
- Inserts contact data into a data extension (aka Journey Entry Event data extension)
- Fire an entry event to trigger the contact into the Journey.
To set up an API entry Event see https://help.salesforce.com/articleView?id=mc_jb_admit_contacts_via_api.htm&type=5. Once you have set up an Entry Event you will be provided with an Event Definition Key.
You will need to pass this key with the JB Fire Entry Event API call. This is the API the Mulesoft team will call.
Host: https://YOUR_SUBDOMAIN.rest.marketingcloudapis.com POST /interaction/v1/events Content-Type: application/json Authorization: Bearer YOUR_ACCESS_TOKEN { "ContactKey": "ID601", "EventDefinitionKey":"AcmeBank-AccountAccessed", "Data": { "accountNumber":"123456", "patronName":"John Smith" } }
Consolidation and transfer to SFMC is happening realtime, as soon as there is abandoned checkout data, it is enriched with "productImage" field and send to SFMC.
Nothing is sent to SFMC if there were no abandoned checkouts for this specific store.
Shopify webhooks spec: https://help.shopify.com/en/api/reference/events/webhook
List of stores: /wiki/spaces/RES/pages/134119897
CDS docs: /wiki/spaces/GUA/pages/326238486
Archive - Objectstore description
ObjectStore - service provided by MuleSoft as a part of CloudHub. It is distributed storage engine - in memory or persistent. Data can be accesed by ObjectStore Mule adapter or by REST API.
Example how to retrievie data from OS by API. To use object store from outside of Mule cloud hub it needs to go through this steps:
1. Get token:
curl --request POST \ --url https://anypoint.mulesoft.com/accounts/login \ --header 'content-type: application/json' \ --data '{ "username": "your login", "password": "your password" }'
2. Get list of environments:
curl --request GET \ --url https://anypoint.mulesoft.com/accounts/api/organizations/8025b860-8aef-4b56-b662-c9fee1c6d5e7/environments \ --header 'authorization: Bearer token-from-step-1'
3. Get list of stores:
curl --request GET \ --url https://anypoint.mulesoft.com/accounts/api/organizations/8025b860-8aef-4b56-b662-c9fee1c6d5e7/environments \ --header 'authorization: Bearer token-from-step-1’
4. Get list of partitions:
curl --request GET \ --url https://object-store-eu-central-1.anypoint.mulesoft.com/api/v1/organizations/8025b860-8aef-4b56-b662-c9fee1c6d5e7/environments/8416a5f6-31de-4886-b2af-abdc6a2d4a8e/stores/NAME OF STORE/partitions \ --header 'authorization: Bearer token-from-step-1'
5. Get list of keys:
curl --request GET \ --url https://object-store-eu-central-1.anypoint.mulesoft.com/api/v1/organizations/8025b860-8aef-4b56-b662-c9fee1c6d5e7/environments/8416a5f6-31de-4886-b2af-abdc6a2d4a8e/stores/NAME OF STORE/partitions/DEFAULT_PARTITION/keys \ --header 'authorization: Bearer token-from-step-1’
6. Get values by key:
curl --request GET \ --url https://object-store-eu-central-1.anypoint.mulesoft.com/api/v1/organizations/8025b860-8aef-4b56-b662-c9fee1c6d5e7/environments/8416a5f6-31de-4886-b2af-abdc6a2d4a8e/stores/NAME OF STORE/partitions/DEFAULT_PARTITION/keys/key-name \ --header 'authorization: Bearer token-from-step-1'
Result will be simillar to this:
{ "stringValue": "value", "keyId": "key-name", "valueType": "STRING" }
Development information:
Please install lombok plugin at Anypoint Studio to get support for auto-generated code(Intellij support through plugin on the market):
- Copy lombok.jar file onto AnypointStudio\plugins
- Add at end of AnypointStudio\AnypointStudio.ini -javaagent:lombok.jar
- Restart IDE
- At info panel "About Anypoint Studio" you shall sth similar to see:
Additional fields table