ICF 1.5.20.28

Multiple CSV connector

The Multiple CSV connector allows for resources such as users and groups to be retrieved from one or more CSV files as defined by the connector configuration. When used continuously in production, CSV files serve as a change log, often containing only records that have changed.

  • This connector does not verify CSV data. You must ensure that your CSV file is complete and properly formed before using the connector.

  • Do not remove or replace CSV files that are the source or target of an active scheduled reconciliation.

  • Modifying the CSV header by adding, removing, or modifying the columns requires reconfiguring the connector for the changes to take effect.

Install the Multiple CSV connector

If you are looking for the Advanced Identity Cloud application for this connector, refer to:

You can download any connector from Backstage, but some are included in the default deployment for Advanced Identity Cloud, IDM, or RCS. When using an included connector, you can skip installing it and move directly to configuration.

Connector included in default deployment
Connector IDM RCS

Multiple CSV

No

No

Download the connector .jar file from Backstage.

  • If you are running the connector locally, place it in the /path/to/openidm/connectors directory, for example:

    mv ~/Downloads/multicsv-connector-1.5.20.28.jar /path/to/openidm/connectors/

  • If you are using a remote connector server (RCS), place it in the /path/to/openicf/connectors directory on the RCS.

Configure the Multiple CSV connector

Create a connector configuration using the IDM admin UI:

  1. From the navigation bar, click Configure > Connectors.

  2. On the Connectors page, click New Connector.

  3. On the New Connector page, type a Connector Name.

  4. From the Connector Type drop-down list, select Multiple CSV Connector - 1.5.20.28.

  5. Complete the Base Connector Details.

    For a list of all configuration properties, refer to Multiple CSV Connector Configuration

  6. Click Save.

When your connector is configured correctly, the connector displays as Active in the admin UI.

Refer to this procedure to create a connector configuration over REST.

{
  "configurationProperties": {
    "minBytesToEstimate": 100000000,
    "multiValuedFieldDelimiter": ";",
    "newLineString": "\n",
    "quotationMode": "ALL",
    "csvFiles": [
      {
        "objectType": "__ACCOUNT__",
        "path": "&{idm.instance.dir}/data/file_name.csv",
        "uid": "_CHANGEME_",
        "name": "_CHANGEME_",
        "password": "_CHANGEME_",
        "multiValuedAttr": [
          "_CHANGEME_"
        ]
      },
      {
        "objectType": "__GROUP__",
        "path": "&{idm.instance.dir}/data/file_name.csv",
        "name": "_CHANGEME_",
        "uid": "_CHANGEME_",
        "multiValuedAttr": [
          "_CHANGEME_"
        ]
      },
      {
        "objectType": "__BALANCE__",
        "path": "&{idm.instance.dir}/data/file_name.csv",
        "name": "_CHANGEME_",
        "uid": "_CHANGEME_"
      }
    ],
    "quoteCharacter": "\"",
    "escapeCharacter": "\\",
    "fieldDelimiter": ",",
    "ignoreSurroundingSpaces": false
  }
}
minBytesToEstimate

The total number of rows in the CSV file will begin to be estimated when the file size in bytes exceeds the value specified in this field. The default value is 100MB.

multiValuedFieldDelimiter

String value that delimits each field inside a multivalued field.

newLineString

Indicates the character that represents a new line in the CSV file.

quotationMode

Defines quoting behavior.

  • Available values:

    • ALL: Quotes all fields.

    • ALL_NON_NULL: Quotes all non-null fields.

    • MINIMAL: Quotes fields that contain special characters such as a field delimiter, quote character, or any of the characters in the line separator string.

    • NON_NUMERIC: Quotes all non-numeric fields.

    • NONE: Never quotes fields.

csvFiles

This field is required. It’s an array of objects, where each object represents the CSV file associated with a corresponding object type.

objectType

The name of the object type.

path

The path where the CSV file is located.

name

The field that represents the name of the record.

password

The field to be handled as a password. It will be hidden and encrypted.

uid

The field that represents the ID of the record

multiValuedAttr

Refers to an array that can contain multiple values for the same row.

quoteCharacter

Defines the characters used to enclose values.

escapeCharacter

Defines the characters used to escape special characters.

fieldDelimiter

String value that delimits each field. Defaults to ",".

ignoreSurroundingSpaces

Ignores whitespace around field separators. Can alter CSV structure. Defaults to false.

The character used for quoteCharacter, escapeCharacter, and fieldDelimiter must be unique.

Multiple CSV remote connector

If you want to run this connector outside of PingOne Advanced Identity Cloud or IDM, you can configure the Multiple CSV connector as a remote connector. Java Connectors installed remotely on a Java Connector Server function identically to those bundled locally within PingOne Advanced Identity Cloud or installed locally on IDM.

You can download the Multiple CSV connector from here.

Refer to Remote connectors for configuring the Multiple CSV remote connector.

Use the Multiple CSV connector

In the following example, we will use account, but it can be replaced by any object type.

If you change the structure of the CSV file resource by adding or removing columns, you must update the corresponding object type in the connector configuration accordingly.

You can use the Multiple CSV connector to perform the following actions.

List all Multiple CSV accounts

This example queries all Multiple CSV accounts:

curl \
--header "X-OpenIDM-Username: openidm-admin" \
--header "X-OpenIDM-Password: openidm-admin" \
--request GET \
"http://localhost:8080/openidm/system/Multiple CSV/__ACCOUNT__?_queryFilter=true"
{
  "result": [
    {
      "_id": "9012",
      "Department": "Sales",
      "First_name": "Rachel",
      "Recovery_code": "rb9012",
      "Phones": [
        "1234",
        "5678"
      ],
      "__NAME__": "rachel@example.com",
      "Last_name": "Booker",
      "Location": "Manchester"
    },
    ...
    {
      "_id": "2070",
      "Department": "Depot",
      "First_name": "Laura",
      "Recovery_code": "lg2070",
      "Phones": [
        "1234",
        "5678"
      ],
      "__NAME__": "laura@example.com",
      "Last_name": "Grey",
      "Location": "London"
    }
  ],
  "resultCount": 10,
  "pagedResultsCookie": null,
  "totalPagedResultsPolicy": "NONE",
  "totalPagedResults": -1,
  "remainingPagedResults": -1
}

The value of the totalPagedResults field is estimated for files larger than 100000000 (100MB).

List Multiple CSV field account by ID

This example queries Multiple CSV file account by ID:

curl \
--header "X-OpenIDM-Username: openidm-admin" \
--header "X-OpenIDM-Password: openidm-admin" \
--request GET \
"http://localhost:8080/openidm/system/Multiple CSV/__ACCOUNT__/9012"
{
  "_id": "9012",
  "Department": "Sales",
  "First_name": "Rachel",
  "Recovery_code": "rb9012",
  "Phones": [
      "1234",
      "5678"
  ],
  "__NAME__": "rachel@example.com",
  "Last_name": "Booker",
  "Location": "Manchester"
}

Create a Multiple CSV file full account

This example creates a Multiple CSV file account with the minimum required attributes.

curl \
--header "X-OpenIDM-Username: openidm-admin" \
--header "X-OpenIDM-Password: openidm-admin" \
--header "Content-Type: application/json" \
--request POST \
--data '{
{
  "__UID__": "7854",
  "__NAME__": "Jane Doe",
  "__PASSWORD__": "12345678",
  "email": "janedoe@example.com"
  "First_name": "Jane",
  "Last_name" : "Doe",
  "Department": "Sales",
  "Location": "London",
  "Phones": [1234,5678]
}' \
"http://localhost:8080/openidm/system/connectorName/__ACCOUNT__?_action=create"
{
  "_id": "859d70a5-573e-4428-8294-1a13a38e0fec",
  "Department": "Sales",
  "First_name": "Jane",
  "Recovery_code": "cj4082",
  "Phones": [
    "123",
    "45678"
  ],
  "__NAME__": "janedoe@example.com",
  "Last_name": "Doe",
  "Location": "London"
}

A record can be created with a specific __UID__, or a random ID is assigned if not specified. When you create a new account you must specify at least the __NAME__ attribute.

Update a Multiple CSV account

The following command updates a specific Multiple CSV account by its ID:

curl \
--header "X-OpenIDM-Username: openidm-admin" \
--header "X-OpenIDM-Password: openidm-admin" \
--header "Content-Type: application/json" \
--header "If-Match: *" \
--request PUT \
--data '{
  "Department": "Engineering",
  "First_name": "Jhon",
  "Phones": [876,54321],
  "__NAME__": "jhondoe@example.com",
  "Last_name": "Doe",
  "Location": "Manchester"
}' \
'http://localhost:8080/openidm/system/Multiple CSV/__ACCOUNT__/859d70a5-573e-4428-8294-1a13a38e0fec'
{
  "_id": "859d70a5-573e-4428-8294-1a13a38e0fec",
  "Department": "Engineering",
  "First_name": "Jhon",
  "Recovery_code": "cj4082",
  "Phones": [
    "876",
    "54321"
  ],
  "__NAME__": "jhondoe@example.com",
  "Last_name": "Doe",
  "Location": "Manchester"
}

Delete a Multiple CSV account

The following example deletes a Multiple CSV account:

curl \
--header "X-OpenIDM-Username: openidm-admin" \
--header "X-OpenIDM-Password: openidm-admin" \
--header 'Content-Type: application/json' \
--request DELETE \
'http://localhost:8080/openidm/system/Multiple CSV/__ACCOUNT__/859d70a5-573e-4428-8294-1a13a38e0fec'
{
  "_id": "859d70a5-573e-4428-8294-1a13a38e0fec",
  "Department": "Engineering",
  "First_name": "Jhon",
  "Recovery_code": "cj4082",
  "Phones": [
    "876",
    "54321"
  ],
  "__NAME__": "jhondoe@example.com",
  "Last_name": "Doe",
  "Location": "Manchester"
}

The response returns the account object before deletion.

OpenICF Interfaces Implemented by the Multi CSV File Connector

The Multi CSV File Connector implements the following OpenICF interfaces. For additional details, see ICF interfaces:

Authenticate

Provides simple authentication with two parameters, presumed to be a user name and password.

Create

Creates an object and its uid.

Delete

Deletes an object, referenced by its uid.

Resolve Username

Resolves an object by its username and returns the uid of the object.

Schema

Describes the object types, operations, and options that the connector supports.

Script on Connector

Enables an application to run a script in the context of the connector.

Any script that runs on the connector has the following characteristics:

  • The script runs in the same execution environment as the connector and has access to all the classes to which the connector has access.

  • The script has access to a connector variable that is equivalent to an initialized instance of the connector. At a minimum, the script can access the connector configuration.

  • The script has access to any script arguments passed in by the application.

Search

Searches the target resource for all objects that match the specified object class and filter.

Test

Tests the connector configuration.

Testing a configuration checks all elements of the environment that are referred to by the configuration are available. For example, the connector might make a physical connection to a host that is specified in the configuration to verify that it exists and that the credentials that are specified in the configuration are valid.

This operation might need to connect to a resource, and, as such, might take some time. Do not invoke this operation too often, such as before every provisioning operation. The test operation is not intended to check that the connector is alive (that is, that its physical connection to the resource has not timed out).

You can invoke the test operation before a connector configuration has been validated.

Update

Updates (modifies or replaces) objects on a target resource.

Multi CSV File Connector Configuration

The Multi CSV File Connector has the following configurable properties:

Configuration properties

Property Type Default Encrypted(1) Required(2)

minBytesToEstimate

double

1.0E8

No

Minimum number of bytes from which the number of records will be estimated. The default value is 1E+8 bytes

spaceReplacementString

String

_

No

The character(s) used to replace spaces within column names.The default value is an underscore (_)

multiValuedFieldDelimiter

String

null

No

The character in the CSV file that is used to separate Multi valued field values. The default value is a semicolon (;)

newLineString

String

\n

No

The character string in the CSV file that is used to terminate each line. The default value is a newline character(" ")

quotationMode

String

ALL

No

Specifies the quotation behavior. For instance, ALL applies quotation to all columns. The default value is "ALL"

quoteCharacter

String

"

No

The character in the CSV file that is used to encapsulate strings. The default value is double quote (")

escapeCharacter

String

\

No

The character in the CSV file that is used to escape characters. The default value is a backslash ("\")

fieldDelimiter

String

,

No

The character in the CSV file that is used to separate field values. The default value is a comma (,)

ignoreSurroundingSpaces

boolean

false

No

True if spaces around values are ignored, false if they are treated as part of the value. The default value is "false"

(1) Whether the property value is considered confidential, and is therefore encrypted in IDM.

(2) A list of operations in this column indicates that the property is required for those operations.