Completeness - AWS Glue
Services or capabilities described in AWS documentation might vary by Region. To see the differences applicable to the AWS European Sovereign Cloud Region, see the AWS European Sovereign Cloud User Guide.

Completeness

Checks the percentage of complete (non-null) values in a column against a given expression.

Syntax

Completeness <COL_NAME> <EXPRESSION>

  • COL_NAME – The name of the column that you want to evaluate the data quality rule against.

    Supported column types: Any column type

  • EXPRESSION – An expression to run against the rule type response in order to produce a Boolean value. For more information, see Expressions.

Example: Null value percentage

The following example rules check if more than 95 percent of the values in a column are complete.

Completeness "First_Name" > 0.95
Completeness "First_Name" > 0.95 where "weightinkgs > 10"

Sample dynamic rules

  • Completeness "colA" between min(last(5)) - 1 and max(last(5)) + 1

  • Completeness "colA" <= avg(last(10))

Null behavior

Note on CSV Data Formats: Blank rows on CSV columns can display multiple behaviors.

  • If a column is of String type, the blank row will be recognized as an empty string and will not fail the Completeness rule.

  • If a column is of another data type like Int, the blank row will be recognized as NULL and will fail the Completeness rule.