Data Factory Copy Activity is failing for Hubspot data source
A Data Factory Copy Activity, sourcing data from a HubSpot connector and sinking to a CSV file in a data lake, is consistently failing with a data type conversion error on the source side.
The pipeline is attempting to copy data without an explicitly defined schema, causing Data Factory to infer data types incorrectly.
Error Code: UserErrorWriteFailedFileOperation
- Root Exception:
Type=System.InvalidCastException - Message:
Value '' for column 'amount' cannot be converted to Double.,Source=Microsoft.DI.Driver.HubSpot - Pipeline Component: Failure is occurring on the 'Source' side during the data conversion/staging process before the write operation.
The failure is specifically tied to the 'amount' column from the HubSpot source.
- Schema Inference: Data Factory's default schema inference samples the HubSpot data and determines the 'amount' column should be a
Double(numeric) type. - Invalid Data: The HubSpot data contains records where the 'amount' field is an empty string (
''). - Conversion Failure: The copy engine attempts to cast the empty string (
'') to the inferredDoubletype, which results in the fatalInvalidCastException. - The following steps were attempted within the Copy Activity settings to resolve the issue, but the UI prevented modification of the inferred source type:
Attempted to manually set the Source Type for the amount column from Double to String in the Copy Activity Mapping tab but the type cannot be edited.
Attempted to clear and re-import the mapping. Failed again.
Attempted to use a json file type as the sync dataset and that failed.