Platform API
Filter And Moderation

Filter and Moderation

To protect your users from harmful language and unsafe content, SendBird provides a set of filters and a variety of moderation methods. The filters are designed to detect with precision anything that is seemingly inappropriate in user-submitted messages, images, and so on. The moderation methods help you reduce the need for human reviews and submission approvals. With the supported filters and moderation methods, you can determine the level of suitability of language and content for your SendBird application.


Domain filter

Filters out text and file messages that contain an inappropriate domain. You can configure which domains to detect and how to process the filtered messages based on your policies and criteria.

HTTP request

Light Color Skin
Copy
PUT https://api-{application_id}.sendbird.com/v3/applications/settings_global

Request body

The following table lists the properties of an HTTP request that this action supports.

Properties
OptionalTypeDescription

domain_filter

nested object

A domain filter configuration on the URLs which are specified in the text and file messages.

domain_filter.domains[]

array

Specifies an array of domains to filter. Each item of the array is specified at least with a combination of domain name and TLD (top level domain) like 'amazon.com'.

domain_filter.type

int

Determines the filtering mode which is applied to the URLs that contain the specified domains. Valid values are 0 (none), 1 (pass if matched), and 2 (block if matched).

Request body example for turning on the filter at an application-wide level
Light Color Skin
Copy
{
    "domain_filter": {
        "domains": ["casinobetting.com", "betonhorserace.com"],
        "type": 2 
    }
}

If you want to turn off the domain filter, send a PUT request with the blank domains property like below:

Request body example for turning off the filter at an application-wide level
Light Color Skin
Copy
{
    "domain_filter": {
        "domains": []
    }
}

Response

If successful, this action returns the updated settings of an application or a custom channel type in the response body.


Profanity filter

Filters out text and file messages that contain a profanity word or pattern. You can configure which words to detect and how to process the filtered messages based on your policies and criteria.

HTTP request

Light Color Skin
Copy
// application-wide settings.
PUT https://api-{application_id}.sendbird.com/v3/applications/settings_global

// settings only applied to channels with a custom channel type.
PUT https://api-{application_id}.sendbird.com/v3/applications/settings_by_channel_custom_type/{custom_type}

Parameters

The following table lists the parameters that this action supports.

Parameters
RequiredTypeDescription

custom_type

string

Specifies the custom channel type to apply a set of settings to channels with the corresponding type.

Request body

The following table lists the properties of an HTTP request that this action supports.

Properties
OptionalTypeDescription

profanity_filter

nested object

A filter configuration on certain words and patterns for matching character combinations in strings, which are not allowed to be used within the application.

profanity_filter.keywords[]

array

Specifies an array of words to filter. *word filters all words that end with "word" including "word" itself while word* filters all words that start with "word" including "word" itself.

profanity_filter.regex_filters[]

list

Specifies a list of regular expressions used for filtering. Each item of the list is specified in {"regex": a pattern in regular expression} format.

profanity_filter.type

int

Determines the filtering method which is applied to profanity words and patterns that match the specified keywords and regular expressions. Acceptable values are limited to 0 (none), 1 (replace), and 2 (block). 1 (replace) replaces only the keywords property with asterisks (*) found in the filtered words. 2 (block) prevents users from sending messages that contain the keywords property or partially match the regex_filters property. (Default: 0)

application-wide
channels-with-custom-type
Light Color Skin
Copy
# Request body example for turning on the filter at an application-wide level 
{
    "profanity_filter": {
        "keywords": "ass,snatch,sperm",
        "regex_filters": [
            {
                "regex": "[^!@#$%^&*]*(fuck|prick)[^!@#$%^&*]*"
            }, 
            {
                "regex": "(http://|https://)?(casino|sex)+([-.]{1}[a-z0-9]+)*.[a-z]{2,5}(:[0-9]{1,5})?(/.*)?"
            }
        ],
        "type": 2   
    }
}
Light Color Skin
Copy
# Request body example for turning on the filter only effective to channels with a custom channel type
{
    "custom_type": "Sports",
    "profanity_filter": {
        "keywords": "idiot,dummy,damn",
        "regex_filters": [
            {
                "regex": "[^!@#$%^&*]*(bitch|assole)[^!@#$%^&*]*"
            }, 
            {
                "regex": "(http://|https://)?(casino|sex|betting)+([-.]{1}[a-z0-9]+)*.[a-z]{2,5}(:[0-9]{1,5})?(/.*)?"
            }
        ],
        "type": 2   
    }
}

If you want to turn off the profanity filter, send a PUT request with the blank keywords and regex_filters properties like below:

application-wide
channels-with-custom-type
Light Color Skin
Copy
# Request body example for turning off the filter at an application-wide level 
{
    "profanity_filter": {
        "keywords": "",
        "regex_filters": []
    }
}
Light Color Skin
Copy
# Request body example for turning off the filter only effective to channels with a custom channel type
{
    "custom_type": "Sports",
    "profanity_filter": {
        "keywords": "",
        "regex_filters": []
    }
}

Response

If successful, this action returns the updated settings of an application or a custom channel type in the response body.


Profanity-triggered moderation

Based on the profanity filter, this feature moderates the users who are sending profanity words to a channel. You can apply different profanity-triggered moderation to appllication-wide and channels with a custom type separately. You can also adjust the level of the moderation by configuring the number of violation limit, the time window for counting violations, and the type of moderation penalty.

HTTP request

Light Color Skin
Copy
// application-wide settings.
PUT https://api-{application_id}.sendbird.com/v3/applications/settings_global

// settings only applied to channels with a custom channel type.
PUT https://api-{application_id}.sendbird.com/v3/applications/settings_by_channel_custom_type/{custom_type}

Parameters

The following table lists the parameters that this action supports.

Parameters
RequiredTypeDescription

custom_type

string

Specifies the custom channel type to apply a set of settings to channels with the corresponding type.

Request body

The following table lists the properties of an HTTP request that this action supports.

Properties
OptionalTypeDescription

profanity_triggered_moderation

nested object

A moderation configuration on which penalty is automatically imposed on users who reach the profanity violation limit within a channel.

profanity_triggered_moderation.count

int

Specifies the number of profanity violation limit which then imposes a penalty on a user if reached. A value of 0 indicates that automatically-triggered moderation is turned off. A value of equal to or larger than 1 indicates that the moderation is turned on and imposes a penalty on a user who commits a number of violations equal to or beyond the set value of the count property within the set time of the duration property.

profanity_triggered_moderation.duration

int

Specifies the duration of the time window in seconds which counts the number of a user’s violations within a channel. For example, if the count property is 2 and the duration property is 5, the number of violations equal to or beyond 2 will be moderated for every 5 seconds. The maximum value is 86400 which indicates 60 days. (Default: 1 second)

profanity_triggered_moderation.action

int

Determines the type of moderation penalty within a channel which is permanently imposed on users until canceled. Valid values are 0 (no action), 1 (mute), 2 (kick), and 3 (ban). (Default: 0)

application-wide
channels-with-custom-type
Light Color Skin
Copy
# Request body example for turning on the moderation at an application-wide level 
{
    "profanity_filter": {
        "keywords": "ass,snatch,sperm",
        "regex_filters": [
            {
                "regex": "[^!@#$%^&*]*(fuck|prick)[^!@#$%^&*]*"
            },
            {
                "regex": "(http://|https://)?(casino|sex)+([-.]{1}[a-z0-9]+)*.[a-z]{2,5}(:[0-9]{1,5})?(/.*)?"
            }
        ],
        "type": 2
    },
    "profanity_triggered_moderation": {
        "count": 5,
        "duration": 5,
        "action": 1
    }
}
Light Color Skin
Copy
# Request body example for turning on the moderation effective to channels with a custom channel type
{
    "custom_type": "Sports",
    "profanity_filter": {
        "keywords": "idiot,dummy,damn",
        "regex_filters": [
            {
                "regex": "[^!@#$%^&*]*(bitch|assole)[^!@#$%^&*]*"
            },
            {
                "regex": "(http://|https://)?(casino|sex|betting)+([-.]{1}[a-z0-9]+)*.[a-z]{2,5}(:[0-9]{1,5})?(/.*)?"
            }
        ],
        "type": 2
    }
    "profanity_triggered_moderation": {
        "count": 2,
        "duration": 10,
        "action": 2 
    }
}

If you want to turn off the profanity-triggered moderation, send a PUT request with the count property of 0 like below:

application-wide
channels-with-custom-type
Light Color Skin
Copy
# Request body example for turning off the moderation at an application-wide level
{
    "profanity_triggered_moderation": {
        "count": 0  
    }
}
Light Color Skin
Copy
# Request body example for turning off the moderation only effective to channels with a custom channel type
{
    "custom_type": "Sports",
    "profanity_triggered_moderation": {
        "count": 0  
    }
}

Response

If successful, this action returns the updated settings of an application or a custom channel type in the response body.


Text moderation

This feature is for moderating the text messages that are perceived as toxic and might have impacts on conversations between your users. You can apply different text moderation settings to appllication-wide and channels with a custom type separately.

Text moderation is one of SendBird's premium features, contact our sales team for further assistance.

HTTP request

Light Color Skin
Copy
// application-wide settings.
PUT https://api-{application_id}.sendbird.com/v3/applications/settings_global

// settings only applied to channels with a custom channel type.
PUT https://api-{application_id}.sendbird.com/v3/applications/settings_by_channel_custom_type/{custom_type}

Parameters

The following table lists the parameters that this action supports.

Parameters
RequiredTypeDescription

custom_type

string

Specifies the custom channel type to apply a set of settings to channels with the corresponding type.

Request body

The following table lists the properties of an HTTP request that this action supports.

Properties
OptionalTypeDescription

text_moderation

nested object

A moderation configuration on which penalty is imposed on abusive and harassing text messages. Perspective API is used for text moderation.

text_moderation.languages[]

array

Specifies an array of one or more language codes specifying the languages which are moderated. Currently, the moderation supports only English (en), Spanish (es), and French (fr). If not specified, the API will automatically detect the text message language.

text_moderation.requestedAttributes

nested object

Specifies a map of one or more attributes which represent machine learning models to score the perceived impact that a text message might have on a conversation in terms of its toxicity probability. Currently, TOXICITY model is only supported for a limited set of languages.

text_moderation.requestedAttributes.(key)

nested object

Specifies a list of one or more score information sets for the attribute. Each score set consists of the scoreThreshold and actions properties to configure when and how to moderate. You can set different moderation standards based on the score, applying different penalties on a message, a user, or a channel.

text_moderation.requestedAttributes.(key).scoreThreshold

float

Specifies a floating-point value in the range [0,1] to only moderate the text messages scoring above the specified value. (Default: 0.9)

text_moderation.requestedattributes.(key).actions

array

Determines the types of moderation penalties which are imposed on a message, a user, or a channel at the same time. Valid values are limited to message_delete, user_ban, user_mute, channel_freeze, and channel_delete. If not specified, no penalty is imposed. (Default: empty array)

text_moderation.moderationLevel

int

Determines the moderation level which is applied to the text messages perceived as toxic. Valid values are 0 (turning off moderation), 1 (manual moderation for reported messages), 2 (automatic moderation for reported messages), and 3 (automatic moderation for all sent messages). (Default: 0)

text_moderation.doNotStore

boolean

Determines whether to allow Perspective API to store text messages for future research to improve the API over time. (Default: true, which doesn't allow the API to store the submitted text messages.)

application-wide
channels-with-custom-type
Light Color Skin
Copy
# Request body example for turning on the moderation at an application-wide level
{
    "text_moderation": {    // For more information, see the 'Properties' table above.
        "languages": ["en"],    // English.
        "requestedAttributes": {
            "TOXICITY": [   // Currently we only support 'TOXICITY'. 
                {
                    "scoreThreshold": 0.7,
                    "actions": ["user_mute"]
                },          
                {
                    "scoreThreshold": 0.9,
                    "actions": ["message_delete", "user_ban", "channel_freeze"] 
                }
            ]
        },
        "moderationLevel": 2, 
        "doNotStore": true
    }
}
Light Color Skin
Copy
# Request body example for turn off the moderation only effective to channels with a custom channel type
{
    "custom_type": "Sports",
    "text_moderation": {    // For more information, see the 'Properties' table above.
        "languages": ["en", "es"],  // English and Spanish
        "requestedAttributes": {
            "TOXICITY": [   // Currently we only support 'TOXICITY'. 
                {
                    "scoreThreshold": 0.8,
                    "actions": ["user_ban", "channel_freeze"] 
                },
                {
                    "scoreThreshold": 1.0,
                    "actions": ["channel_delete"]
                },
            ]
        },
        "moderationLevel": 2, 
        "doNotStore": false 
    }
}

If you want to turn off the text moderation, send a PUT request with the moderationLevel property of 0 like below:

application-wide
channels-with-custom-type
Light Color Skin
Copy
# Request body example for turning off the moderation at an application-wide level
{
    "text_moderation": {
        "moderationLevel": 0
    }
}
Light Color Skin
Copy
# Request body example for turning off the moderation only effective to channels with a custom channel type
{
    "custom_type": "Sports",
    "text_moderation": {
        "moderationLevel": 0
    }
}

Response

If successful, this action returns the updated settings of an application or a custom channel type in the response body.


Image moderation

SendBird’s image moderation feature is powered by Google Cloud Vision API. This feature is for moderating the text and file messages with explicit images or inappropriate image URLs, and uses five categories such as adult, spoof, medical, violence, and racy. You can apply different text moderation settings to appllication-wide and channels with a custom type separately.

Image moderation is one of SendBird's premium features, contact our sales team for further assistance.

HTTP request

Light Color Skin
Copy
// application-wide settings.
PUT https://api-{application_id}.sendbird.com/v3/applications/settings_global

// settings only applied to channels with a custom channel type.
PUT https://api-{application_id}.sendbird.com/v3/applications/settings_by_channel_custom_type/{custom_type}

Parameters

The following table lists the parameters that this action supports.

Parameters
RequiredTypeDescription

custom_type

string

Specifies the custom channel type to apply a set of settings to channels with the corresponding type.

Request body

The following table lists the properties of an HTTP request that this action supports.

Properties
OptionalTypeDescription

image_moderation

nested object

A moderation configuration on inappropriate images within the application. Google Cloud Vision API is used for image moderation and supports many types of images.

image_moderation.type

int

Determines the moderation method which is applied to the images and image URLs in the text and file messages. Valid values are limited to 0 (none), 1 (normal), and 2 (strict). A value of 0 indicates that no moderation is imposed. A value of 1 indicates that the messages with images or image URLs are blocked if the images or image URLs violate the content policies. A value of 2 indicates that the file messages with no images are also blocked in addition to the messages with explicit images or inappropriate image URLs.

image_moderation.soft_block

boolean

If true, the moderation method set by the type property above is ignored and no moderation is imposed on the text and file messages in regard to explicit images or inappropriate image URLs. It will only give the image analysis results in the response. If false, the image moderation works according to the moderation method already set.

image_moderation.limits

nested object

A set of features pertaining to the images for moderation. Each feature specifies the content likelihood for the image which is used as a moderation standard. Acceptable likelihood values are 0 (unknown), 1 (very unlikely), 2 (unlikely), 3 (possible), 4 (likely), and 5 (very likely).

image_moderation.limits.adult

int

Specifies the adult content likelihood for the image.

image_moderation.limits.spoof

int

Specifies spoof likelihood.

image_moderation.limits.medical

int

Specifies likelihood that the image is a medical image.

image_moderation.limits.violence

int

Specifies likelihood that the image contains violent content.

image_moderation.limits.racy

int

Specifies likelihood that the image contains racy content.

image_moderation.check_urls

boolean

Determines whether to check if the image URLs in the text and file messages are appropriate.

application-wide
channels-with-custom-type
Light Color Skin
Copy
# Request body example for turning on the moderation at an application-wide level 
{
    "image_moderation": {   // For more information, see the 'Properties' table above.
        "type": 1,  // 'normal' moderation method.
        "soft_block": false,
        "limits": {
            "adult": 3,
            "spoof": 5,
            "medical": 5,
            "violence": 3,
            "racy": 4
        },
        "check_urls": true  // Check if the image URLs in messages are appropriate.
    }
}
Light Color Skin
Copy
# Request body example for turn off the moderation only effective to channels with a custom channel type
{
    "custom_type": "Sports",    
    "image_moderation": {   // For more information, see the 'Properties' table above.
        "type": 2,  // 'strict' moderation method.
        "soft_block": false,
        "limits": {
            "adult": 4,
            "spoof": 5,
            "medical": 5,
            "violence": 5,
            "racy": 5 
        },
        "check_urls": true  // Check if the image URLs in messages are appropriate.
    }
}

If you want to turn off the image moderation, send a PUT request with the type property of 0 like below:

application-wide
channels-with-custom-type
Light Color Skin
Copy
# Request body example for turning off the moderation at an application-wide level
{
    "image_moderation": {
        "type": 0
    }
}
Light Color Skin
Copy
# Request body example for turning off the moderation only effective to channels with a custom channel type
{
    "custom_type": "Sports",
    "image_moderation": {
        "type": 0
    }
}

When a message has passed through image moderation, SendBird server sends back a success response body like the following:

application-wide
channels-with-custom-type
Light Color Skin
Copy
# Response body example when a message has passed through image moderation
{
    ...
    "moderation_action": 1,
    "moderation_info": {
        "action": 1,
        "message": "Success",
        "scores": { // Image analysis results.
            "adult": false,
            "spoof": false,
            "medical": false,
            "violence": false,
            "racy": false
        },
        "cache_hit": false
    },
    ...
}
Light Color Skin
Copy
# Response body example when a message has passed through image moderation
{
    "custom_type": "Sports",
    ...
    "moderation_action": 2,
    "moderation_info": {
        "action": 2,
        "message": "Success",
        "scores": { // Image analysis results.
            "adult": false,
            "spoof": false,
            "medical": false,
            "violence": false,
            "racy": false
        },
        "cache_hit": false
    },
    ...
}

When a message has been blocked by image moderation, SendBird server sends back an error response containing the 900066 (ERROR_FILE_MOD_BLOCK) or 900065 (ERROR_FILE_URL_BLOCK) code.

Response

If successful, this action returns the updated settings of an application or a custom channel type in the response body.